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INTRODUCTION 


It  is  generally  agreed  that  regular  screening  by  mammography  is  a  woman's  best  strategy  for 
preventing  death  due  to  breast  cancer.  However,  mammography  quality  is  of  concern  for 
three  reasons.  First,  recent  evidence  of  variability  in  radiologists’  interpretations  of  the  same 
mammograms  suggests  that  improvement  is  needed  in  mammographers’  accuracy  in  reading 
films.'  Second,  growing  attention  to  issues  of  costs  and  cost-effectiveness  suggests  the 
importance  of  improving  specificity  in  reading  mammograms.^’^  Third,  the  efficacy  of 
screening  younger  women  remains  controversial.^  '’’^’*’’ 

The  purpose  of  this  project  is  to  demonstrate  that  accuracy  in  film  interpretation  can  be 
significantly  improved  by  a  continuous  quality  improvement  (CQI)  program  which  provides 
mammographers  with  immediate  feedback  on  their  interpretations  of  films  specifically 
selected  for  their  teaching  value. 

The  Fred  Hutchinson  Cancer  Research  Center  (FHCRC),  the  Department  of  Radiology  at  the 
University  of  Washington  (UW),  and  the  Washington  State  Cancer  Registry  (WSCR)  at  the 
Department  of  Health  (DOH)  are  collaborating  to  develop  and  implement  a  Mammography 
Quality  Improvement  Program  (MQIP),  in  order  to  demonstrate  its  feasibility  and 
effectiveness  for  dissemination.  The  MQIP  emphasizes  improvement  in  film  interpretation, 
within  the  context  of  a  comprehensive  program  designed  to  meet  the  requirements  of  the 
Mammography  Quality  Standards  Act  (MQSA)  of  1994. 

The  MQIP  employs  routine  systematic  monitoring  of  measurable  outcomes  of  screening 
mammography,  including  sensitivity,  specificity,  and  positive  predictive  value.  This  is 
referred  to  as  its  surveillance  function.  It  also  identifies  for  mammographers  their  false 
positive  and  false  negative  cases,  so  that  they  can  improve  quality  through  review  of  their  own 
films.  This  is  its  audit  function.  In  addition,  it  provides  continuing  education  for  radiologists, 
and  training  for  technologists,  as  required  by  MQSA.  This  is  its  certification  function.  Most 
importantly,  it  incorporates  immediate  feedback  following  a  radiologist’s  interpretation  of 
practice  films  selected  for  their  educational  value.  This  is  its  continuous  quality  improvement 
(CQI)  function.  The  MQIP  is  comprehensive,  and  will  ensure  that  participating  facilities  are  in 
compliance  with  evolving  accreditation  rules. 

The  primary  objective  of  this  project  is  to  develop  and  implement  a  comprehensive, 
continuous  MQIP  in  16  mammography  facilities  located  in  rural  areas  of  Washington  State. 
The  demonstration  of  the  MQIP  involves  development  and  implementation  of  the  4  functions 
noted  above. 

A  research  study  is  being  conducted  within  the  demonstration  project.  The  primary  research 
objective  is  to  determine  if  the  CQI  program  can  increase  the  accuracy  with  which 
mammographers  interpret  films.  Secondary  research  objectives  are  to  1)  determine  inter¬ 
rater  variability  in  film  interpretation  in  a  set  of  films  selected  for  their  teaching  value,  before 
and  after  implementation  of  the  CQI  program;  2)  determine  post-CQI  intra-rater  variability  in 
film  interpretation;  3)  determine  if  digitized  films  can  be  interpreted  with  the  same  accuracy 
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as  can  high-quality  copies  of  films;  and  4)  determine  if  the  accuracy  with  which  films  are 
interpreted  depends  on  covariates,  the  age  of  the  woman  being  of  particular  interest.  The 
availability  of  comparison  films  will  also  be  considered  as  a  covariate. 

This  four  year  project  is  currently  at  the  end  of  its  first  year. 

BODY 

In  order  to  achieve  project  goals,  18  major  tasks  needing  to  be  accomplished  were  identified 
in  the  original  statement  of  work.  During  year  01  of  the  project,  major  headway  was  made  on 
several  of  these  tasks. 

The  design  of  the  CQI  is  the  core  of  the  MQIP.  It  is  imperative  that  all  aspects  of  the  CQI  be 
carefully  constructed.  To  this  end,  a  manuscript  describing  the  design  of  the  study  and  the 
analysis  of  results  has  been  accepted  for  publication  and  is  included  as  Appendix  A. 

To  implement  the  CQI,  research  was  first  conducted  to  identify  16  mammography  facilities 
meeting  project  criteria.  Each  facility  must  be  located  in  a  rural  area  and  must  not  be 
associated  with  any  other  participating  facility.  In  addition,  facilities  were  selected  so  that 
their  combination  provided  at  least  60  radiologists  who  would  be  eligible  for  participation  in 
the  CQI. 

Once  facilities  were  identified,  recruitment  materials  and  procedures  were  developed  to  solicit 
the  participation  of  these  facilities  and  their  radiologists  in  the  project.  Detailed  recruitment 
procedures  as  well  as  materials  used  in  the  recruitment  process  are  attached  as  Appendix  B. 
The  recruitment  plan  was  begun  in  the  latter  half  of  the  first  year  of  funding  and  will  continue 
into  year  02.  In  addition,  to  ensure  the  participation  of  radiologists,  CME  credit  for 
participation  in  the  CQI  was  obtained  through  the  University  of  Washington. 

Additional  work  for  the  CQI  portion  of  the  MQIP  has  involved  the  development  of  software 
designed  to  present  digitized  mammograms  presented  on  a  personal  computer.  The  software 
allows  the  radiologist  to  indicate  the  location  of  any  potential  malignancy,  and  select  an 
assessment  of  the  mammogram.  The  software  tracks  each  assessment  made  by  the 
radiologist,  scores  the  entire  test  session,  and  provides  feedback  for  incorrectly  read 
mammograms.  At  the  end  of  year  01,  the  software  is  95%  complete  and  in  the  debugging 
phase. 

To  perform  the  CQI,  180  mammogram  studies  meeting  project  criteria  must  be  obtained, 
copied,  and  digitized.  Mammogram  studies  that  can  be  included  in  this  study  must  be 
challenging,  but  interpretable.  Approximately  half  will  contain  a  malignancy.  Specific 
procedures  for  selecting,  obtaining,  and  handling  films  were  developed  during  the  first  year 
and  the  process  of  selecting  the  films  has  begun.  To  obtain  such  a  large  amount  of  films 
meeting  project  criteria,  it  has  been  necessary  to  solicit  the  help  of  community  radiologists.  A 
great  deal  of  time  had  been  invested  in  contacting  radiologists  and  assisting  them  with  film 
selection.  To  date,  approximately  60  studies  have  been  identified. 
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Once  all  film  studies  are  identified,  digitized,  and  entered  in  to  the  CQI  software,  the  process 
of  conducting  a  CQI  session  will  be  piloted  on  a  test  facility.  A  mammography  facility 
located  in  a  rural  area  has  been  identified  and  recruited  to  participate  as  the  pilot  site. 
Currently,  project  staff  are  in  the  process  of  linking  this  facility  to  the  Mammography  Tumor 
Registry  (MTR).  The  MTR  was  established  three  years  ago  as  a  registry  of  mammography 
data  from  local  facilities  linked  to  Washington  State  and  SEER  Cancer  Registry  data.  The 
purpose  of  the  MTR  is  to  provide  surveillance  as  well  as  feedback  to  participating  facilities 
and  radiologists. 

To  address  the  surveillance  component  of  the  MQIP,  work  has  been  done  to  modify  the  MTR 
to  allow  for  the  inclusion  of  facilities  that  do  not  use  a  computerized  system  to  store  their 
mammography  data.  A  paper  worksheet  has  been  developed  and  will  be  presented  to  paper- 
based  facilities  in  hopes  of  convincing  them  to  use  this  form.  If  a  facility  prefers  to  use  their 
own  form,  modifications  will  be  made  to  the  MTR  as  necessary  to  accommodate  that  facility. 

The  MTR  database  itself  has  undergone  several  structural  changes  over  the  course  of  year  01 
in  preparation  for  working  with  the  MQIP  facilities.  The  procedures  for  obtaining,  validating 
and  linking  data  and  report  generation  have  been  modified  to  work  in  a  more  efficient  manner 
thereby  allowing  the  participation  of  a  greater  number  of  facilities  in  the  registry.  Current 
work  with  the  CQI  pilot  facility  is  testing  the  new  MTR  procedures. 

The  certification  component  of  the  MQIP  requires  continuing  education  for  mammography 
technologists  as  well  as  the  radiologists.  To  meet  this  requirement,  research  has  begun  to 
determine  the  appropriate  format  and  agenda  for  the  technologist  training.  To  this  end,  a 
focus  group  with  mammography  technologists  located  at  the  MQIP  pilot  facility  was 
conducted.  Technologists  participating  in  this  focus  group  described  their  vision  of  a  useful 
training  session  that  would  include  hands  on  training,  information  on  breast  pathology, 
training  for  special  situations  such  as  a  handicapped  patient,  and  MQSA  requirements  among 
other  topics.  The  first  technologist  training  event  will  occur  in  year  02  of  the  project. 

Another  training  session  that  will  be  conducted  as  part  of  the  MQIP  is  for  state  Certified 
Tumor  Registrars  (CTR’s).  The  purpose  of  this  training  is  to  focus  on  their  retrieval  of  data 
from  medical  records  and  to  increase  the  quality  of  the  data,  specifically  the  recording  of 
TNM  staging  and  extent  of  disease  information.  It  is  anticipated  that  this  training  will 
improve  the  quality  of  the  cancer  data  that  is  linked  to  the  MTR  and  will  thereby  improve  the 
content  of  the  reports  generated  for  the  MQIP  radiologists  by  providing  more  reliable  and 
detailed  size,  stage  and  grade  data.  Preparation  for  this  training  event  has  been  underway  and 
it  will  happen  in  November,  1997. 

CONCLUSIONS 

Year  01  of  the  project  has  primarily  been  spent  preparing  for  the  implementation  of  the 
various  components  that  make  up  the  MQIP.  Because  the  project  is  still  in  its  initial  phases, 
there  are  few  conclusions  to  present. 
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Work  that  has  been  completed  includes  the  design  of  the  CQI  as  described  in  the  paper  in 
Appendix  A.  The  design  of  the  study  incorporates  measuring  sensitivity  and  specificity  rather 
than  ROC  curves.  The  statistical  analysis  that  was  involved  with  designing  the  CQI  study 
demonstrates  that  sensitivity  and  specificity  are  much  better  measures  of  performance  for 
mammography,  especially  using  the  universally  accepted  BIRADS  coding  scheme.  This 
work  is  a  significant  contribution  to  the  field  of  mammography  quality  and  the  ongoing 
discussion  of  the  validity  of  ROC  curves  for  measuring  mammography  and  performance. 
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Design  of  a  Study  to  Improve  Accuracy  in  Reading 

Mammograms 

ABSTRACT.  This  paper  is  concerned  with  the  design  and  ^alysis  of 
mammoffraphy  reading  studies.  In  particular  we  consider  studies  aimed  at 

evaluatiS  intLventions  to  improve  the  axxur^y  with  “fX'JSe^roup  of 

read.  A  simple  randomized  design  is  suggested  in  which  a  relatively  large  g  p 

readers  read  sets  of  mammograms  before  and  after  ^  of 

propose  solutions  to  three  difficult  statistical  issues 

such  studies;  (i)  the  choice  of  primary  outcome  measure;  (u)  the  data  analysis 
technique  to  bVemployed;  and  (iii)  the  methodology  for  calculating  sample  sizes  for 

readers  and  images  to  be  read. 

First,  we  argue  in  favor  of  using  sensitivity  and  specifidty^  the 
measure  rather  than  receiver  operating  characteristic 

ktter  axe  considered  state  of  the  art  for  many  types  of  raffiolop  reding  studies. 

We  argue  that  sensitivity  and  specificity  axe  more  chnically  relevant  and 
conceptually  more  straightforward  than  ROC  curves.  Second,  we  suggest  a 
bivaxi^e  approach  to  data  analysis  for  evaluating  inte^ention  effects  on  sensitivi  y 
and  specifidty.  This  accommodates  the  correlations  inherent  between 
measmes  and  allows  for  estimation  of  joint  effects  on  them.  Finally  we  propose  a 
method  for  power  calculations  which  utilizes  computer  simffiation  t^iques. 

Simple  formulas  for  sample  size  calculations  axe  not  a\mlaWe  in  part  becaus 
variability  in  accuracy  amongst  readers  and  variation 

introduce  complexity  into  power  calculations.  The  simulatip  method  which  w 
propose  accommodates  such  complexity  and  is  easy  to  implement. 

The  methodology  was  motivated  by  a  study  funded  by  the  Depaxtinent  of  ^e^e 
to  evaluate  the^itential  efficacy  of  an  edu^cational 

this  study  we  illustrate  the  steps  involved  m  power  cal(^ation8  ^d  fPP^y 
analytic  techniques  to  the  sort  of  data  expected  to  result  from  this  study.  Though 
the  proposed  methods  were  motivated  by  this  particular  study,  the  statistu^ 
considerations  axe  relevant  more  broadly  in  mammography  and  md^  in 
types  of  radiologic  imaging  studies.  Standards  te  the  condurt  of  radiologic  re^ 
iSdies  are  not  yet  weU  developed,  as  they  are  for  randomized  dimcal  tnals  md  for 
case<ontrol  studies.  We  hope  that  the  discussion  in  this  paper  mU  add  to  the 
dialogue  necessary  for  development  of  such  standards. _ _ 

KEY  WORDS.  ROC  curves,  sensitivity  and  specificity,  computer  simulation, 
diagnostic  tests,  screening. 
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1.  INTRODUCTION 


Mammography  screening  for  breast  cancer  has  been  shown  to  be  associated  with 
decreased  breast  cancer  mortality,  at  least  in  women  over  the  age  of  50  years  [1]. 
Major  efforts  are  currently  underway  to  improve  participation  by  women  m  screen¬ 
ing  programs  [2].  Nevertheless,  there  is  concern  about  the  quality  of  mammography 
screening  and  there  is  general  agreement  that  improvements  in  quality  may  lead  to 
improvements  in  the  performance  of  mammography  as  a  screening  modality.  Quality 
might  be  improved  for  example  by  impro\dng  the  imaging  procedures.  Alternatively, 
improvements  in  the  accuracy  with  which  mammographers  interpret  mammograms 
may  improve  the  performance  of  screening  mammography.  Recent  studies  [3,4]  have 
shown  that  there  is  considerable  variability  amongst  radiologists  m  their  interpreta¬ 
tions  of  screening  mammograms.  Elmore  et  al  [3]  observed  that  sensitiidties  ranged 
from  74%  to  96%  and  that  spedfidties  ranged  from  35%  to  89%  among  10  radiolo¬ 
gists  reading  150  selected  mammograms.  Beam  et  al  [4]  using  a  much  larger  sample 
of  108  radiologists,  eadi  reading  79  mammograms,  found  sensitivities  in  the  range 
of  47%  to  100%  and  spedficities  in  the  range  of  35%  to  99%.  These  observations 
suggest  that  improvement  in  interpretation  may  be  possible. 

As  part  of  a  project  called  the  Mammography  Quality  Improvement  Project 
(MQIP)  funded  by  the  Department  of  Defense  and  aimed  at  improving  the  quality 
of  mammography  screening  in  rural  communities,  we  are  developing  an  educational 
program  to  improve  the  accuraqr  with  which  radiolo^sts  interpret  mammograms. 
The  educational  intervention  is  comprised  of  a  series  of  five  sessions  in  whidi  mam¬ 
mographers  read  films  and  axe  provided  with  immediate  feedback  on  the  accuracy  of 
their  interpretations.  Feedback  is  provided  using  a  laptop  personal  computer  which  is 
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mailed  to  the  radiologist  prior  to  Us  reading  session.  The  computer  program  empha¬ 
sizes  the  particular  features  of  ea*  mammogram  which  are  relevant  to  determining 
the  disease  status  of  the  woman  screened.  Eventually  it  may  be  possible  to  dissem 
mate  this  sort  of  intervention  over  computer  networks  thus  making  it  attractive  m 
terms  of  easy  accessibility  and  low  cost. 

In  order  to  evaluate  the  impact  of  such  an  intervention  on  improvements  in  di- 
agnostic  accurao-  it  will  eventually  be  necessary  to  perform  a  study  of  radiologists 
interpretations  of  screening  mammograms  in  their  actual  practices.  As  a  prdimi- 
nary  step  to  sudi  a  large-scale  study,  we  wBl  evaluate  the  intervention  effects  in  a 
more  controlled  setting.  Specifically,  we  wiU  have  a  number  of  radiologists  read  a 
selected  set  of  mammograms  before  and  after  the  intervention  and  evaluate  changes 
in  accuracy.  The  mammograms  included  in  tUs  contmUed  study  Will  be  comprised 
of  about  50%  from  women  with  disease,  a  proportion  much  larger  than  would  be 
observed  in  practice  but  necessarily  Ugh  in  order  to  estimate  sensitivity  rates  in  a 
small-scale  study.  Mammograms  wiU  be  selected  to  represent  a  reasonably  broad 

range  of  interpretive  difficulty. 

The  purpose  of  this  paper  is  to  elucidate  some  of  the  key  statistical  issues  in  the 
design  of  such  a  controlled  reading  study.  Standards  for  the  design  of  such  studies  are 
not  weU  developed.  This  contrasts  with  therapeutic  cUnical  trials  and  epidemiologic 
studies  where  the  basic  elements  of  study  design  are  now  fairly  well  standardized  [5]. 
The  question  we  propose  to  address  in  this  reading  study,  namely  evaluation  of  an 
intervention  effect  in  a  controlled  setting,  is  a  standard  sort  of  question  addressed 
in  diagnostic  imaging  research.  Hence  the  design  issues  which  are  dealt  with  here 
will  have  implications  for  future  studies  in  mammography  and  in  other  diagnostic 


00013 


test  settings.  These  same  issues  also  arise  in  reading  studies  designed  to  compare 
different  imaging  modalities.  The  key  issues  concern  the  choice  of  relevant  primary 
outcome  measures,  appropriate  data  analysis  strategies,  and  methodology  for  power 
calculations  which  incorporates  variability  amongst  radiologists  and  amongst  images. 
Broader  issues  in  regards  to  study  designs  for  evaluating  imaging  tests  have  been 
discussed  in  a  more  general  sense  in  the  literature  [6,7j. 

In  Section  2  we  consider  two  sets  of  measures  which  can  be  used  to  define  accu¬ 
racy  in  reading  mammograms;  firstly,  sensitivity  and  spedfidty  and  secondly,  ROC 
curves.  We  argue  in  favor  of  the  former,  in  part,  because  they  are  more  clinically 
relevant  and  most  easily  understood  but  also  because  the  latter  can  provide  inappro¬ 
priate  conclusions  concerning  intervention  benefits.  In  Section  3  we  detail  the  basic 
elements  of  the  statistical  design  of  our  study  which  could  be  considered  a  prototype 
for  evaluating  intervention  effects  in  diagnostic  radiology.  An  approach  to  jomt  anal¬ 
ysis  of  sensitivity  and  spedfidty  is  outhned  in  Section  4.  In  Section  5  we  descnbe 
methodology  for  power  calculations  which  are  appropriate  for  the  proposed  design 
and  analysis.  We  propose  the  use  of  computer  simulation  methods  for  calculating 
power  because  they  allow  for  complex  designs  and  can  easily  incorporate  variability 
amongst  radiolopsts  and  images.  Having  described  the  steps  involved  in  calculating 

;'.V 

power  in  Section  5,  we  then  apply  these  procedures  to  the  proposed  MQIP  study  m 
Section  6,  in  order  to  iUustrate  the  methods.  Concluding  remarks  follow  in  Section 

7. 

2.  MEASURES  OF  ACCURACY 
2.1  Definitions 

A  radiologist  reading  a  set  of  mammograms  for  a  woman  in  our  study  will  classify  each 
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breast  according  to  his  or  her  suspicion  of  its  showing  malignancy.  The  ACR  lexicon 
for  rating  a  breast  (8]  which  we  will  employ,  defines  a  5-point  scale  with  category  1 
indicating  “normal,  routine  follow-up  recommended”,  2  indicating  “benign,  routine 
follow-up”,  3  indicating  “probably  benign,  early  recaU  recommended”,  4  indicating 
“suspidous  for  cancer,  consider  biopsy”  and  5  indicating  “highly  suspidous  for  cancer, 
biopsy  recommended”.  A  common  definition  of  a  screen  positive  mammogram  is  one 
which  receives  a  rating  of  4  or  greater.  These  are  mammograms  which  are  sufficiently 
suspidous  for  cancer  that  biopsy  is  recommended  and  hence  they  have  an  impact  on 
dinical  practice.  Sometimes  a  rating  of  3  or  greater  is  considered  positive.  Because 
of  the  clinical  implications  of  ratings  4  and  5,  we  will  focus  on  the  positidty  criterion 

of  cat^ory  >  4  here. 

Given  a  definition  for  screen  positivity,  since  there  is  a  rating  for  each  breast, 
one  can  calculate  sensitivities  and  spedficities  with  either  ‘woman’  or  ‘breast’  as 
the  unit  of  analysis.  The  latter  includes  aU  non-diseased  breasts,  (including  non- 
diseased  breasts  from  women  with  can<»r),  as  the  denominator  for  spedficity  and 
all  diseased  breasts  as  the  denominator  for  sensitivity.  Breast  level  definitions  are 
arguably  inappropriate  in  clinical  studies  of  mammography.  It  seems  more  clinicaUy 
relevant  to  use  woman  as  the  unit  of  analysis.  For  this,  one  could  use  the  maxunum 
of  the  ratings  for  the  left  and  right  sides  as  the  woman  level  rating  for  calculation 
of  sensitivity  and  spedficity.  However,  occasionally  a  woman  with  unilateral  disease 
may  not  have  it  detected  in  the  affected  side  but  will  have  a  positive  mammogram 
on  the  unaffected  side.  In  this  case,  using  the  maximum  rating  will  inappropriately 
inflate  the  sensitivity.  We  define  sensitivity  instead  as  the  proportion  of  women  with 
disease  who  have  it  detected  (a  rating  of  >  4)  on  the  affected  side.  The  specificity  is 
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the  proportion  of  women  without  disease  who  have  a  maximum  rating  of  less  than  4. 

ROC  analysis  is  a  statistical  technique  used  to  describe  accuracy  of  diagnostic 
tests  when  the  test  outcome  is  either  ordinal  or  continuous  as  opposed  to  binary.  The 
rating  data  generated  in  radiology  reading  studies  is  ordinal  and  ROC  analysis  is  often 
considered  optimal  for  the  analysis  of  sudi  studies  as  is  evidenced,  for  example,  in  a 
recent  issue  of  Academic  Radiology  [9].  An  ROC  curve  is  constructed  by  varying  the 
criterion  used  for  defining  a  positive  mammogram  from  “rating  >  2”  to  “rating  >  5”, 
plotting  the  assodated  sensitivity  and  1-spedficity  values  against  each  other,  and 
finally  fitting  a  curve  to  the  points  so  that  the  curve  is  anchored  at  (0,0)  and  (1,1). 
Various  algorithms  exist  for  fitting  a  curve,  the  most  notable  being  the  Dorfman- 
Alf  algorithm  based  on  the  binormal  model  [10]  and  the  empmcal  nonparametric 
method  which  simply  connects  observed  ROC  points  Unearly.  The  area  under  the 
ROC  curve  is  usually  used  to  summarize  accuracy.  Again  we  suggest  that  woman 
rather  than  breast  should  be  the  unit  of  analysis  in  defining  the  ROC  curve.  That  is, 
in  calculating  the  sensitivity  corresponding  to  the  criterion  “rating  >  K  ,  it  should 
be  defined  as  the  proportion  of  women  with  cancer  who  have  a  rating  of  >  A  on  an 

affected  side. 

2,2  ROC  Analysis  Versus  Sensitivity  and  Specificity  ■ 

ROC  analysis  was  developed  ori^nally  for  diagnostic  tests  with  results  on  some  arbi¬ 
trary  scale.  Its  primary  advantage  is  that  it  allows  one  to  assess  the  inherent  capaaty 
of  the  test  to  distinguish  between  diseased  and  non-diseased  subjects  without  linking 
the  test  to  some  particular  threshold  for  defining  screen  positive  [11,12].  This  seems 
appropriate  in  radiology  experiments  when  image  ratings  are  arbitrary  numbers  with 
no  specific  clinical  meaning  attached  to  them.  In  that  case,  shifts  in  the  distribu- 
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tions  of  ratings  are  of  no  consequence  as  long  as  they  are  equally  shifted  for  diseased 
and  non-diseased  subjects.  In  mammography,  however,  mammogram  ratings  have 
very  specific  clinical  meanings  and  consequent  clinical  implications.  Uniform  shifts 
in  the  frequencies  with  whidi  rating  categories  are  chosen  can  have  major  chnical 

implications. 

Moreover,  in  contrast  to  the  prototype  setting  for  ROC  analysis,  shifts  between 
certain  diagnostic  categories  are  of  more  importance  than  others.  For  example,  as 
noted  by  Kopans  (13],  whether  an  image  is  rated  in  category  4  versus  category  5 
has  no  clinical  impact.  Similarly  classifications  in  category  1  versus  category  2  are 
clinically  irrelevant.  However,  shifts  between  categories  4  or  5  and  between  1  or  2 
can  have  a  big  impact  on  the  ROC  analysis.  To  illustrate  this  consider  the  setting 
shown  in  Figure  1.  The  effect  of  intervention  in  this  setting  is  to  shift  classifications  of 
diseased  observations  from  category  4  to  category  5  and  classification  of  non-diseased 
subjects  from  category  2  to  cat^ory  1.  Though  these  changes  are  of  no  climcal 
import,  the  ROC  type  analysis  indicates  a  benefit  for  the  intervention.  Thus  an 
ROC  analysis  can  indicate  a  benefit  of  intervention  even  though  a  climcally  relevant 
benefit  does  not  exist. 

Of  even  more  concern  is  the  fact  that  a  clinically  relevant  benefit  of  interven- 
tion  can  occur  even  when  the  ROC  curves  pre-  and  post-intervention  axe  the  same. 
Consider  the  ROC  curve  depicted  in  Figure  2  for  such  a  situation.  The  location 
on  the  ROC  curve  of  the  points  associated  with  the  criterion  “rating  >  category 
4”  indicate  that  sensitivity  was  significantly  increased  without  decreasing  specificity. 
This  clinically  relevant  improvement  in  test  accuracy  does  not  manifest  itself  in  an 
improvement  in  the  ROC  curves  since  the  pre-and  post-intervention  curves  axe  the 
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same.  (Interestingly,  classic  binormal  ROC  curves  do  not  fit  the  situation  depicted  in 
Figure  2  and  a  binormal  ROC  analysis  in  this  setting  may  incorrectly  indicate  that 
the  ROC  curve  post-intervention  is  improved  over  that  pre-intervention). 

The  fact  that  ROC  analysis  can  yield  inappropriate  conclusions  r^arding  the 
clinically  relevant  effects  of  intervention  argues  against  its  use  for  the  pnmary  analysis 
of  mammography  reading  study  data.  Another  valid  argument  for  not  using  an  ROC 
analysis  is  that  it  is  complicated  and  not  easily  understood  by  cliniaans.  Moreover, 
the  so-called  ‘area  under  the  curve’  which  summarizes  the  ROC  curve  in  a  single 
number  has  an  interpretation  which  is  not  wdl  known  or  easily  understood.  It  can  be 
interpreted  as  the  probability  that  a  radiologist  will  have  a  greater  suspicion  of  cancer 
from  a  mammogram  from  a  woman  with  disease  than  from  a  woman  without  [14]. 
This  probability,  however,  seems  to  be  of  more  theoretical  than  practical  relevance. 

We  propose  using  the  more  clinically  meaningful  quantities  of  sensitivity  and 
spedfidty  for  the  primary  data  analysis  and  employing  ROC  analysis  as  a  secondary 
descriptive  device.  Though  ROC  analysis  may  be  statistically  more  powerful  in  some 
settings,  any  study  should  be  designed  so  that  it  has  adequate  power  to  detect  changes 
in  the  quantities  which  are  of  practical  relevance.  Hence,  we  suggest  that  power 
calculations  for  a  mammography  reading  study  should  be  based  on  th^^  ability  to 
detect  changes  in  sensitivity  and  specifidty  rather  than  on  the  basis  of  detecting 
changes  in  ROC  curves. 

3.  STUDY  DESIGN 

We  now  describe  the  basic  elements  of  the  design  which  we  propose  for  studies  eval¬ 
uating  intervention  effects  on  reading  accuraqr  in  mammography.  In  this  prototype 
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design,  radiologists  are  randomly  assigned  to  intervention  and  control  groups,  with 
the  number  in  the  former  bdng  denoted  by  Rt  and  the  number  in  the  latter  denoted 
by  Rc-  Two  image  sets  are  constructed  with  M  images  in  each  set  5  =  1,2.  In  set 
5,  a  number  M^,  are  from  women  with  disease  and  this  number  may  differ  between 
the  two  sets.  Each  reader  reads  one  set  of  images  before  the  intervention  penod  and 
one  set  after.  It  is  important  that  the  sets  before  and  after  intervention  be  differ¬ 
ent  since  readers  may  remember,  to  some  degree,  images  that  they  have  previously 
read.  Half  of  the  readers  chosen  at  random  in  each  of  the  intervention  and  control 
groups  read  set  1  before  intervention  and  set  2  after  intervention.  The  other  half 
read  them  in  the  opposite  order:  set  2  followed  by  set  1.  This  cross-over  of  film  sets 
eliminates  the  possibUity  of  systematic  bias  due  to  film  sets.  The  design  is  balanced 
in  the  sense  that  set  1  is  read  equally  often  before  and  after  the  intervention  phase 
in  both  the  intervention  and  control  groups,  and  similarly  for  set  2.  Readers  are  told 
the  approximate  prevalence  of  diseased  images,  i.e.,  (JWp  -f  Af^)/2M  and  that  this 
varies  between  the  two  sets.  The  rationale  for  telling  the  readers  the  approximate 
prevalence  is  that  it  will  become  apparent  in  any  case  after  reading  the  first  set  of 
images  and  that  apriori  knowledge  of  it  should  reduce  the  potential  impact  as  much 
as  possible  on  the  observed  improvement  in  accuracy.  Readers  will  use  the  ACR 
lexicon  to  classify  mammograms  and  for  each  reading  it  will  be  determined  if  it  is 
screen  positive  or  negative  according  to  whether  the  rating  is  greater  than  3  or  not. 

Images  for  inclusion  in  the  study  need  to  be  selected  so  that  average  sensitivity 
and  specifidty  at  the  baseline  assessment  are  relatively  low.  That  is,  improvements 
in  accuracy  should  be  possible  with  the  sets  of  images  chosen.  K,  in  the  absence  of 
intervention  all  images  from  women  with  disease  were  easily  identified  as  such,  the 
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observed  sensitivities  pre-  and  post-intervention  would  be  close  to  1  and  a  change 
in  sensitivity  would  not  be  identifiable  regardless  of  the  actual  effect  of  intervention. 
Thus  at  least  some  of  the  diseased  images  should  be  difficult  but  not  impossible 
to  identify  as  being  from  women  with  disease.  Analogous  considerations  apply  to 
specificity  and  the  choice  of  non-diseased  images  included  in  the  study. 


4.  DATA  ANALYSIS 


Having  described  the  basic  elements  of  the  design  and  the  choice  of  primary  out¬ 
comes,  we  turn  now  to  the  strategy  for  data  analysis.  There  are  two  components  to 
the  analysis.  The  first  concerns  a  comparison  of  post-  versus  pre-intervention  reading 
accuracy  among  the  Rt  readers  in  the  intervention  group.  The  second  is  the  compar¬ 
ison  of  changes  from  pre-  to  post-intervention  between  the  intervention  and  control 
groups.  We  first  consider  the  former  analysis,  in  part  because  it  allows  us  to  define 
notation  most  easily. 


The  purpose  of  this  data  analysis  is  to  compare  the  overall  sensitivity  pre-intervention 
with  that  post-intervention  and  to  compare  the  overall  specificity  pre-intervention 
with  that  post-intervention.  K  5r^re  and  5rj)ost  denote  the  observed  pre-  and  post- 
intervention  sensitivities  for  radiolo^st  r,  then  the  observed  change  in  the  overall 
sensitivity  A(sensitivity)  is  the  average  change  in  sensitivities  across  radiblo^sts  in 
the  intervention  group: 

1  ^  ^  \ 

Ar(sensitivity)  =  —  ^  (5r,post  —  ^r,pre  j  • 

^Tr=l 

Similarly  the  observed  change  in  the  overall  specifidty  in  the  intervention  group  is 

1  ^  ^  \ 

Arfspecifidty)  =  22  (■^r.post  “ 

-n-T  r=l 
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where  and  denote  the  observed  pre-  and  post-intervention  speafiaties 

for  radiologist  r.  Variance  estimators  for  AT(sensitivity)  and  ArCspecifiaty)  are  pro¬ 
vided  in  the  appendix.  Although  Ar(sensitivity)  and  At  (specificity)  axe  sample 
means  of  changes  in  sensitivities  and  specificities,  their  variances  axe  not  given  by  the 
usual  vaxiance  formulae  for  sample  means.  Indeed  such  sample  variances  would  over¬ 
estimate  the  variability.  Rather  the  correct  variance  estimators  rely  on  acknowledging 
that  there  are  in  essence  two  strata  of  radiologists  in  the  design,  which  are  defined  by 
the  ordering  of  the  two  image  sets  which  axe  rated.  The  vanances  of  Ar(sensitivity) 
and  ArCspecificity)  are  averages  of  stratum-specific  variances,  as  shown  in  the  ap¬ 
pendix. 

Sensitivity  and  specificity  axe  highly  correlated  parameters.  Radiologists  with 
high  sensitivities  tend  to  have  low  spedficities.  This  will  happen  for  example  if  they 
have  a  low  threshold  for  classifying  images  as  diseased.  Similarly,  changes  m  sen¬ 
sitivities  and  specificities  induced  by  the  intervention  may  be  Wghly  correlated.  In 
particular,  if  the  intervention  simply  changes  the  implidt  threshold  a  radiologist  has 
for  classifying  a  mammogram  as  diseased  then  the  sensitivity  and  specifidty  wiU 
both  be  changed  but  in  opposite  directions.  Thus  it  is  important  to  assess  joint 
effects  of  intervention  on  sensitivity  and  specifidty  and  to  account  for  correlations 
between  them  in  making  inference.  This  can  be  accomplished  by  employing  a  bi¬ 
variate  analysis  approach  which  is  a  spedal  case  of  multivariate  analysis,  and  for 
which  thore  is  a  large  statistical  literature  (15].  Using  this  approach  to  test  the 
hypotheses  that  the  true  average  sensitivity  and  specifidty  are  unchanged  by  the 
intervention.  Ho  :  A(sensitivity)  =  A(specificity)  =  0,  a  chi-squared  test  statistic  is 
calculated.  This  statistic  is  a  function  of  the  observed  average  changes,  A(sensitivity) 
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and  A(spedficity),  their  variances  and  also  their  correlation.  An  expression  for  the 
chi-squared  statistic  is  provided  in  the  appendix* 

In  addition  to  simply  testing  the  hypothesis  of  no  intervention  effect,  it  will  be 
important  to  provide  a  confidence  region  for  the  intervention  effects  on  sensitivity 
and  specificity  based  on  the  observed  data.  That  is,  a  range  of  intervention  effects, 
{A(sensitivity),  A(specificity)},  which  are  consistent  with  the  observed  data.  Such  a 
joint  95%  confidence  region  is  defined  formally  as  the  set  of  values  (x,y)  for  which 
the  hypothesis  Ho  :  {A(sensitivity)  =  x,  A(specificity)  =  y}  is  not  rejected  at  the 
5%  significance  level.  This  r^ion  is  an  elipse,  centered  at  the  observed  intervention 
effect  (A(sensitivity),  A(specificity)).  We  refer  the  interested  reader  to  the  text 
[15]  by  Johnson  and  Wichern  (1988,  section  5.2)  for  technical  details  regarding  its 
calculation.  Code  for  calculating  such  regions  has  been  written  by  Murdoch  and 
Chow  for  the  S-PLUS  statistical  software  package  and  can  be  obtained  from  the  S- 
archive  on  the  Statlib  computer  site  (http://lib.stat.cmu.edu).  In  a  similar  fashion 
a  joint  confidence  region  for  the  overall  average  sensitivity  and  spedficity  pre-  or 
post-intervention  can  be  calculated.  It  is  calculated  using  the  observed  radiologist 
specific  sOTsitivities  and  spedficities  pre-  and  post-intervention,  and  requires  only 
calculation  of  the  means,  variances  and  correlations  for  these  parameters.  In  order 
to  illustrate  these  analyses.  Figure  3  displays  joint  confidence  re^ons  based  on  a 
simulated  data  set.  In  our  opinion  these  confidence  regions  provide  a  simple  summary 
of  the  information  contained  in  study  data  regarding  intervention  effects  on  reading 
accuracy.  In  the  simulated  data,  the  analyses  show  that  sensitivity  was  increased  by 
the  intervention  whereas  there  is  no  evidence  of  change  in  specifidty. 

So  far  we  have  considered  the  comparison  of  post-  versus  pre-intervention  reading 
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ziccuracy  within  the  intervention  group.  In  order  to  attribute  changes  in  accuracy  to 
the  intervention  it  will  be  necessary  to  compare  the  changes  in  the  intervention  group 
with  those  in  the  control  group.  Without  the  control  group  comparison  observed 
changes  might  be  attributed  to  other  factors,  such  as  the  increased  reading  practice  or 
increased  awareness  of  reader  fallibility  induced  by  participation  in  the  study.  Thus, 
turning  now  to  the  comparison  of  intervention  and  control  groups,  the  main  hypoth¬ 
esis  to  be  tested  is  that  the  changes  in  sensitivity  and  spedficity  in  the  intervention 
group  are  the  same  as  those  in  the  control  group.  Using  a  subscript  T  to  denote  the 
intervention  group  and  subscript  C  to  denote  the  control  group,  the  null  hypothesis 
is  H  :o  Ac(sensitivity)  =  Ax(sensitivity),  Ac(specificity)  =  Ar(specifidty).  A  test 
statistic  which  has  a  chi-squared  distribution  with  2  degrees  of  freedom  is  described 
in  the  appendix  for  testing  this  hypothesis.  Joint  confidence  regions  for  the  differ¬ 
ences  in  changes  between  the  groups,  namely  Ar(sensitivity)  —  Ac(sensitivity)  and 
ArCspecificity)  -  Ac(specifidty),  can  be  calculated  using  methods  analogous  to  those 
described  earlier  for  the  pre-versus-post-intervention  comparison. 

5.  METHODOLOGY  FOR  POWER  CALCULATIONS 

Power  calculations  for  the  reading  study  are  somewhat  complicated.  They  must 
accommodate  the  facts  that  readers  vary  in  their  accuracy  parameters  of  sensitivity 
and  specificity,  that  their  sensitivities  and  spedficities  are  likdy  negatively  correlated, 
that  images  vary  in  difficulty  and  that  a  bivariate  analysis  approadi  will  be  employed. 
These  factors  together  make  analytic  expressions  for  sample  size  intractable.  We  in¬ 
stead  take  a  computer  simulation  approadi  to  power  calculations.  The  simulation 
approach  to  power  calculation  is  a  general  and  standard  method  and  indeed  software 
has  been  developed  for  certain  types  of  applications  (16].  The  basic  idea  is  to  re- 


00023 


peatedly  simulate  data  as  it  is  expected  or  hoped  to  arise  in  the  course  of  the  study, 
and  determine  how  often  the  null  hypothesis  is  rejected.  By  definition  the  statistical 
power  of  the  study  is  the  proportion  of  simulated  studies  in  which  the  null  hypothesis 
is  rejected.  One  calculates  the  power  in  this  fashion  using  various  sample  sizes  until  a 
sample  size  is  found  which  provides  adequate  power.  This  indirect  computer  intensive 
approach  to  sample  size  calculation  is  easily  accomplished  with  modem  computers. 

5.1  Models  for  Pre-  and  Post-intervention  Accuracy 

In  order  to  simulate  study  data  we  need  to  define  precisely  the  mechanisms  giving  nse 
to  the  data.  We  therefore  need  to  make  assumptions  about  the  reading  accuracies 
before  and  after  intervention.  For  this  purpose  we  suppose  that  before  intervention  a 
reader  correctly  assesses  a  woman  with  tumor  as  being  diseased  with  probability  P®-. 
The  probability  P^  depends  on  the  image  denoted  by  i  and  on  the  reader,  denoted 
by  r.  The  probabihties  P®-  will  presumably  be  higher  if  the  tumor  is  clearly  visible 
in  image  i  than  if  it  is  not.  The  probabihties  will  also  be  higher  if  the  radiologist 
is  conservative  and  is  inchned  to  recommend  biopsy  for  borderhne  cases.  We  let 
5^  be  the  sensitivity  of  the  average  radiologist  to  the  average  film  from  a  woman 
with  tumor.  The  variabihty  amongst  films  in  terms  of  the  difficulty  which  readers 
have  in  assessing  them,  is  captured  by  specifying  a  distribution  for  the  sensitivities 
that  the  average  reader  has  in  assessing  the  films.  Here  we  assume  that  the  average 
reader’s  sensitivity  to  films  varies  uniformly  in  an  interval  (5^  -  a®,  5®  +  a®)  across 
different  films.  Thus  for  the  average  radiologist,  easier  films  axe  read  with  sensitivity 
closer  to  +  o,^  and  more  difficult  films  are  read  with  sensitivity  closer  to  —  a,  . 
In  a  similar  fashion,  on  the  average  film  from  a  diseased  woman,  the  sensitivity  of 
different  readers  is  assumed  to  vary  uniformly  in  an  interval  {S^  -  6^,  +  iP) 
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across  radiologists.  Thus  radiologists  with  high  sensitivity  to  the  average  film  will 
have  sensitivity  closer  to  5®  +  hP.  In  the  appendix  we  detail  a  logistic  model  with 
random  effects  (also  called  a  mixed  model)  for  the  probabilities  which  give  rise 
to  inter-image  and  inter-reader  variabiUty  as  postulated  here.  It  is  assumed  that  on 
the  logistic  scale  there  are  no  interactions  between  reader  and  image  specific  effects 
on  the  sensitivity. 

Observe  that  for  the  purposes  of  simulating  data,  by  specifying  and  we  can 
now  generate  a  random  image  effect  by  choosing  a  random  number  in  (5®±a^)  which 
corresponds  to  the  sensitivity  an  average  radiologist  has  for  detecting  it.  Similarly, 
having  a  specified  and  iP  we  are  in  a  position  to  generate  a  random  reader 
effect  by  choosing  a  random  number  in  (5®  -  6°,  +  bP)  which  corresponds  to  his 

sensitivity  to  the  average  film.  The  logistic  model  displayed  in  the  appendix  then 
yields  the  probabiHty  Pi, r  which  that  reader  has  of  correctly  assessing  that  image  as 

diseased. 

Analogous  considerations  apply  to  the  detemunation  of  randomly  generated  spea- 
ficities  which  vary  across  radiologists  and  across  images  from  women  without  disease. 
Values  for  parameters  5^,  iP  and  need  to  be  specified  in  order  to  define  the 
data  generating  process.  Here,  is  the  probabiHty  that  the  average  radiologist  will 
correctly  assess  the  average  non-diseased  image  as  sudi,  radiologists  vary  uniformly 
in  (5®  -  +  bP)  in  their  specificities  to  the  average  non-diseased  film,  and 

images  from  women  without  disease  vary  uniformly  in  {S^  -  aP,SP  +  aP)  in  the 
probabiHties  of  the  average  reader  correctly  classifying  them.  The  sensitivities  and 
spedfidties  from  single  radiologists  should  be  correlated.  In  the  appendix  we  describe 
how  negative  correlation  between  sensitivities  and  specifidties  withm  radiolo^sts  can 
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be  built  into  the  data  simulation  mechanism. 

In  summary,  for  each  study  radiologist  we  simulate  his/her  sensitivity  and  sped- 
ficity  to  the  average  diseased  and  non-diseased  films  respectively,  by  randomly  sam- 
pling  correlated  numbers  from  (S®  -  i”,  S"  +  »“)  and  (S®  -  4®,  S®  +  4®),  respec- 
tively.  For  each  study  film  we  detemune  the  sensitivity  or  spedficity  that  an  average 
radiologist  has  for  it  by  randomly  sampling  a  number  from  -  aP^S^  +  a^)  or 
(S®  -  aP,S^  +  aPy  Finally,  for  each  combination  of  film  i  and  radiologist  r,  we 
can  calculate  PP  or  PP,  which  is  the  probability  that  the  radiologist  will  assess  that 
image  correctly. 

The  Pfl  and  PP^  pertain  to  probabilities  before  intervention  in  the  treatment  and 
control  groups.  One  also  needs  to  specify  treatment  effects  in  order  that  correspond¬ 
ing  probabilities  after  intervention  can  be  calculated.  We  postulate  that  after  inter¬ 
vention  the  quantities  5^  and  5^  are  changed  to  new  values  but  that  the  variations 
amongst  readers  and  amongst  images  remain  the  same.  In  the  appendix  we  define  in 
a  mathematically  predse  way  a  logistic  model  which  incorporates  sudi  intervention 

effects. 

5.2  Simulated  Study  Data  Generation 

Having  spedfied  statistical  models  for  pre-  and  post-intervention  rating  probabiUties 
which  incorporate  variation  amongst  radiologists  and  amongst  images,  we  now  turn  to 
the  simulation  of  study  data  in  accordance  with  the  study  design  which  we  proposed  in 
section  3.  The  first  step  is  to  generate  images  and  image  sets.  This  entails  generating 
M  diseased  images  (i.e.,  M  image-specific  parameters,  one  for  each  image),  generating 
M  non-diseased  images,  and  finally  from  the  2M  films  choosing  M  at  random  without 
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replacement  to  form  film  set  1.  The  remaining  M  hlms  constitute  film  set  2.  The 
next  step  is  to  generate  Rt  intervention  readers  and  Rc  control  readers  and  assign 
them  film  sets.  That  is,  for  each  oiRr-^-Rc  readers  we  generate  pairs  of  pre-  and 
post-intervention  sensitivities  and  specificities  to  average  diseased  and  non-diseased 
films  according  to  the  models  described  in  section  5.1.  Of  the  total  Rt  +  Rc  readers, 
Rt  are  assigned  at  random  to  the  intervention  group  and  the  remaining  Rc  to  the 
control  group.  Finally  film  set  orderings  are  assigned  to  the  readers  with  half  of  the 
intervention  readers  selected  at  random  being  assigned  set  1  first  and  the  other  half 
assigned  set  2  first.  Similarly,  Rcl2  control  readers  axe  assigned  set  1  Mowed  by  set 
2  and  the  other  Rcrl2  readers  are  assigned  film  sets  in  the  opposite  order. 

The  final  step  in  generating  data  for  a  simulated  study  is  to  actually  generate 
the  readings  for  each  reader  and  image  combination.  That  is,  for  each  reader  and 
for  each  of  the  M  films  in  his/her  pre-intervention  set,  a  binary  random  variable  is 
generated  which  is  his/her  assessment  of  whether  or  not  that  image  shows  disease 
using  the  probability  is  diseased  and  1  -  if  the  image  is  not 

diseased.  Similarly,  for  each  of  the  M  films  in  Ws/her  post-intervention  set  a  similar 
binary  random  variable  is  generated  using  Pr,;,post  or  1  —  Prji,post  rioting  that  the 
pre-  and  post-probabilities  differ  by  different  amounts  for  intervention-versus-control 

radiologists. 

Having  generated  the  simulated  study  data  the  test  statistics  of  interest  can  now 
be  calculated.  Data  are  simulated  (first  the  probabiUties,  then  the  ratings)  and 
results  calculated  under  the  same  assumptions  and  study  design  many  times,  with 
1000  or  5000  simulated  datasets  being  typical  numbers  used  for  power  calculations. 
The  proportion  of  simulated  stupes  in  which  the  null  hypothesis  is  rejected  is  the 
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calculated  study  power  for  that  design  and  under  those  assumptions. 

6.  POWER  CALCULATIONS:  RESULTS  FOR  THE  MQIP  STUDY 

To  fix  ideas,  we  now  illustrate  the  computer  simulation  method  for  power  calculations 
in  the  MQIP  study.  This  illustration  also  identifies  some  sources  of  data  to  guide 
assumptions  for  power  calculations. 

We  need  to  choose  assumed  parameters  for  the  baseline  sensitivities  and  specifici¬ 
ties,  for  the  variations  amongst  radiologists  and  amongst  images  and  for  intervention 
effects  of  interest.  We  assume  that  the  median  sensitivity  pre-intervention,  in 
our  study  will  be  in  the  range  of  .70  to  .80.  This  accords  with  previous  studies  which 
found  median  sensitivities  of  .70  and  .80  [3,4].  Median  pre-intervention  specificity 
will  also  be  assumed  to  lie  in  the  range  of  .70  to  .80.  Beam  et  al  [4]  found  a  median 
specificity  of  0.94  for  mammograms  from  women  with  normal  mammograms  and  a 
median  specificity  of  0.60  for  mammograms  from  women  with  benign  disease.  Elmore 
et  al  [3]  found  a  median  specificity  of  0.94.  In  contrast  to  these  studies  we  will  mform 
the  radiologists  of  the  average  prevalence  which  is  higher  than  that  expected  in  a 
practical  screening  setting.  Because  of  this  and  the  fact  that  the  films  in  pur  study 
will  be  somewhat  difficult,  we  anticipate  an  initial  spedficity  lower  than  observed 
in  those  studies.  The  variation  amongst  radiologists  in  sensitivities  and  spedfidties 
will  be  assumed  such  that  iP  =  0.20  and  6®  =  0.20,  which  is  in  agreement  with  the 
range  of  approximately  40%  in  sensitivities  (and  spedficities)  amongst  radiologists 
observed  in  Beam’s  study.  We  could  find  no  data  on  inter-image  variability  to  sug¬ 
gest  appropriate  values  for  o®  and  a^.  We  assume  that  they  are  of  the  same  order 
of  magnitude  as  the  inter-rater  variability  parameters,  aP  =  aP  =  .20.  In  regards 
to  intervention  effects  of  interest,  we  consider  that  changes  of  10  percentage  pomts 
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in  either  sensitivity  or  specificity  are  of  interest.  However,  we  calculated  power  for  a 
variety  of  intervention  effects. 

Practical  considerations  concerning  time  and  cost  dictate  the  range  of  sample 
sizes  which  are  feasible  and  therefore,  for  which  power  calculations  are  performed. 
We  anticipate  that  no  more  than  approximately  80  radiologists  are  available  for  the 
reading  study  in  the  rural  communities  in  which  our  mammography  quality  improve¬ 
ment  study  is  being  conducted.  To  maximize  power,  equal  numbers  of  radiologists 
are  assigned  to  control  and  intervention  groups.  Therefore  the  number  of  radiologists 
per  group  to  be  considered  for  power  calculation  purposes  will  be  in  the  range  of  20 
to  40,  Experience  suggests  that  readers  can  comfortably  read  no  more  than  45  films 
per  session.  We  therefore  calculated  power  for  experiments  in  which  the  number  of 
films  per  set,  M,  was  either  30  or  45. 

Estimates  of  power  based  on  computer  simulations  are  shown  in  Table  1.  Though 
results  are  shown  only  for  intervention  effects  on  sensitivity  with  no  effect  on  sped- 
ficity,  because  of  the  symmetry  inherent  in  the  design,  the  same  power  calculations 
hold  for  a  10%  change  in  specifidty  with  no  change  in  the  sensitivity.  Observe  that 
the  power  is  far  larger  for  the  within  intervention  group  assessment  of  change  than  for 
the  between  group  comparison  of  change.  This  is  to  be  expected  since  the  viability 
involved  in  comparing  two  random  changes  is  greater  than  the  variability  involved  in 
comparing  a  single  change  with  the  null  hypothesis  of  no  diange.  We  also  observe 
from  Table  1  that  the  power  is  less  when  the  baseline  sensitivity  is  0.70  than  when  it 
is  0.80,  This  is  due  to  the  relatively  larger  binomial  variance  for  the  lower  baseline 
rate.  In  order  to  be  conservative  we  focus  on  this  lower  rate.  Interestingly  the  base¬ 
line  specifidty  had  little  impact  on  the  power  to  detect  an  intervention  effect  on  the 
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sensitivity. 


The  target  power  for  our  study  design  is  90%  which  allows  a  10%  chance  of  an 
inconclusive  result  when  the  intervention  increases  sensitivity  from  .70  to  .80.  For 
the  within  intervention  group  comparison  this  cannot  be  achieved  with  20  readers 
but  can  be  achieved  with  30  readers  if  45  images  are  included  in  each  image  set.  The 
between  group  comparison,  however,  has  a  power  of  only  66%  in  this  case.  Even  with 
use  of  our  maximum  resources,  i.e.,  40  readers  per  group  and  45  images  per  reading 
set,  the  power  is  only  80%.  This  allows  for  a  20%  chance  of  an  inconclusive  result 
even  when  there  is  a  clinically  important  intervention  effect  on  diagnostic  accuracy. 

For  the  MQIP  study  we  chose  to  focus  the  study  on  the  within  group  companson. 
The  power  calculations  were  an  important  contribution  to  this  decision  but  other 
considerations  also  played  a  role.  Radiologists  would  have  little  motivation  to  partic¬ 
ipate  in  the  control  arm  Whereas  they  would  receive  CME  credit  for  participation  in 
the  intervention  arm.  The  possibihty  that  those  in  the  control  arm  would  learn  from 
the  baseline  assessment  was  also  a  concern  and  thus  we  were  concerned  that  it  might 
not  even  be  feasible  to  construct  a  true  control  group.  Finally,  it  was  felt  that  if  we 
found  a  definite  positive  change  in  the  intervention  group,  then  this  would  provide 
sufficient  motivation  to  proceed  with  more  comprehensive  controlled  studies  in  the 
future.  Thus  we  chose  to  study  only  the  intervention  effects  in  the  intervention  group 
and  to  use  sample  sizes  of  30  radiologists  each  reading  sets  of  mammograms  from  45 
women  before  and  after  intervention- 

The  simulation  program  allowed  us  the  flexibility  to  explore  the  performance  of 
this  study  design  in  a  variety  of  settings  other  than  that  assumed  for  the  primary 
sample  size  calculation.  First  we  calculated  the  probability  of  rejecting  the  null 
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hypothesis  for  settings  where  there  was  no  intervention  effect.  Recall  that  inference 
for  the  test  statistic  is  based  on  a  chi-squared  statistic  and  is  theoretically  valid 
with  large  samples.  However,  this  study  entails  relatively  small  samples.  We  used 
the  simulations  to  chedc  the  adequacy  of  the  large  sample  theory  in  our  study.  To 
do  this  we  generated  data  under  the  null  hypothesis.  The  rejection  probability  was 
approMmately  .06  in  the  settings  we  studied,  indicating  that  the  true  significance  level 
of  the  test  is  slightly  higher  than  the  target  of  .05  but  adequate  for  our  purposes. 

We  next  explored  the  power  of  this  study  design  and  sample  sizes  to  detect  an 
array  of  intervention  effects.  Results  are  shown  in  Table  2.  Although  the  study  has 
adequate  power  to  detect  a  change  in  sensitivity  (or  spedficity)  of  0.10  even  when 
the  pre-intervention  sensitivity  is  as  low  as  0.60,  it  has  little  chance  of  detecting  a 
smaller  change  of  0.05.  On  the  other  hand,  if  small  changes  of  the  order  of  .05  occur 
in  both  the  average  sensitivity  and  in  the  average  specificity  there  is  a  good  chance 
that  the  simultaneous  effects  will  be  detected. 

7.  DISCUSSION 

Diagnostic  imaging  technology  is  already  a  basic  component  of  medical  care  and  con¬ 
tinues  to  develop  at  a  rapid  pace.  It  is  clearly  unportant  to  assess  the  accuracy  with 
which  readers  can  diagnose  disease  using  such  technologies,  to  evaluate  the  effects 
of  training  strategies  and  to  compare  methods.  Implications  for  public  health  can 
be  enormous.  Unfortunately,  statistical  methodology  for  evaluating  and  comparing 
imaging  methods  has  not  received  much  attention  by  biostatisticians  and  epidemi¬ 
ologists  involved  in  public  health  research.  Rather  the  literature  is  concentrated  in 
radiology  research  journals,  has  generally  focmsed  on  small  scjale  studies  involving 
only  a  few  readers  and  has  ignored  clinical  implications  associated  with  different  di- 
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agnostic  categories.  We  believe  that  it  is  time  to  bring  the  discussion  about  study 
design  and  analysis  for  evaluating  imaging  technology  to  the  broader  community  of 
epidemiologists  and  statisticians  involved  in  public  health.  This  is  particularly  im¬ 
portant  as  interest  increases  in  the  accuracies  and  costs  of  these  imaging  methods. 
By  presenting  our  thoughts  on  the  design  and  analysis  of  a  study  to  evaluate  an  ed¬ 
ucational  intervention  on  the  interpretation  of  mammograms,  we  hope  to  stimulate 
such  discussion. 

The  choice  of  primary  outcome  measure  is  the  most  basic  element  of  any  study 
design.  We  diose  to  consider  the  sensitivity  and  specificity  as  the  basis  for  evaluating 
intervention  effects.  This  conflicts  with  initial  statistical  reviewers  of  our  study  de¬ 
sign  who  were  of  the  opinion  that  ROC  analysis  was  the  only  appropriate  and  indeed 
the  state-of-the-art  basis  for  evaluating  an  intervention  effect.  We  now  argue  that  in 
mammography  where  specific  cUnical  actions  are  associated  with  diagnostic  rating 
categories,  sensitivity  and  specifidty  provide  a  more  clinically  relevant  and  concep¬ 
tually  straightforward  basis  for  comparison  than  does  ROC  analysis.  Moreover  this 
approach  allows  us  to  evaluate  effects  on  false  positive  as  well  as  true  positive  rates. 
In  contrast  ROC  analysis  does  not  quantify  the  false  positive  rates  directly  but  in 
a  sense  only  uses  it  to  standardize  the  true  positive  rate.  We  do  not  dismiss  ROC 
analysis  eatirely  but  rather  we  regard  the  analysis  of  the  specific  rating  categories  of 
secondary  importance  and  focus  the  design  on  sensitivity  and  specificity.  Thus  the 
MQIP  study  was  designed  to  ensure  adequate  power  to  detect  changes  in  the  most 
clinically  relevant  quantities. 

We  also  needed  to  decide  upon  the  analysis  techniques  for  maldng  statistical 
inference  about  sensitivity  and  specificity.  We  propose  to  simultaneously  estimate 
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sensitivity  and  specificity  using  multivariate  methods.  Sensitivity  and  specificity  as 
we  have  defined  them  are  average  sensitivities  and  average  specificities  of  radiologists 
in  our  study.  They  can  also  be  interpreted  as  marginal  or  population  average  quan¬ 
tities,  in  the  sense  of  being  the  probability  that  a  diseased  (or  non-diseased)  image 
will  be  correctly  interpreted  as  such  in  the  study.  The  distinction  between  the  popu¬ 
lation  average  and  average  radiologist-specific  interpretations  has  to  do  with  whether 
one  considers  the  accuracy  parameters  to  be  based  on  data  pooled  across  radiologists 
(population  average)  or  to  be  based  on  calculation  of  the  accuracy  parameter  for  each 
radiologist  and  then  averaging  the  results.  In  our  study  these  quantities  coincide  be¬ 
cause  all  radiologists  expect  to  read  the  same  numbers  of  films.  In  studies  where  this 
is  not  the  case,  the  distinction  should  be  considered  and  a  decision  should  be  made 
regarding  which  of  the  two  entities  is  most  relevant. 

The  approadi  we  propose  for  statistical  inference  is  relatively  straightforward, 
being  based  on  methods  for  inference  about  sample  means.  Confidence  intervals 
are  based  on  the  variance-covariance  matrix  of  the  estimated  (sensitivity,  specifidty) 
parameters  or  their  changes  amongst  radiologists.  Possible  non-normality  of  the 
average  estimates  may  be  an  issue  in  our  study,  though  for  the  settings  considered  in 
the  power  calculation  this  did  not  appear  to  be  the  case.  An  alternative  approach  to 
inference  which  might  be  more  robust  would  follow  the  marginal  regression  modeling 
approach  described  by  Leisenring,  Pepe  and  Longton  [17].  One  could  formulate 
logistic  regression  models  for  the  population  average  sensitivity  and  1-specifiaty  as 

logit  {Prob[screen  positive  |  image  diseased]}  =  7o  + 
logit  {Prob[screen  positive  |  image  non-diseased]}  =  J?o  +  Vi^ 

where  the  logit  function  is  lo^t{x}  =  ln{x/{l  -  x)}  and  b  is  0  if  the  image  was  read 
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before  the  intervention  and  1  if  it  was  read  after  the  intervention.  The  changes  in 
the  true  and  false  positive  rates  are  now  quantified  in  the  odds  ratio  parameters  71 
and  rji,  respectively,  and  joint  confidence  intervals  can  be  calculated.  By  adding  an 
interaction  term  between  b  and  /,  where  I  is  an  indicator  of  the  radiologist  being  m 
the  control  or  intervention  groups: 

logit  {Prob[screen  positive  |  image  diseased]}  =  70  +  7i^  +  72^ 
logit  {Prob[screen  positive  ]  image  non-diseased]}  =  170  +  Vi^  + 

a  comparison  of  the  changes  in  the  intervention  and  control  groups  can  be  made  by 
testing  if  the  parameters  72  or  1)2  are  0.  Though  this  logistic  regression  modelling 
approach  may  provide  more  robust  confidence  intervals,  we  felt  that  the  simpler 
approach  described  earlier  was  adequate  for  power  calculations. 

The  prototype  reading  study  we  have  described  concerns  evaluating  the  effect  of 
an  intervention  on  the  change  in  accuracy  parameters.  We  note,  however,  that  most  of 
our  discussion  is  also  relevant  to  the  comparison  of  accuracies  associated  with  different 
imaging  modalities.  Suppose  for  example,  that  there  are  two  sets  of  women  (denoted 
by  set  1  and  set  2)  from  which  images  have  been  made  using  two  modalities.  A 
natural  study  design  to  compare  the  modalities  would  entail  readers  assigned  to  read 
one  set  of  films  produced  with  one  modality  and  the  other  set  of  films  produced  with 
the  other  modaUty.  Using  the  notation  1(A)  to  denote  set  1  produced  with  modality 
A  and  similarly  for  the  other  combination,  readers  read  either  {1(A)  and  2{B)}  or 
{2(A)  and  1(B)}.  Considering  that  the  ordering  may  also  influence  accuracy  param¬ 
eters,  this  yields  four  groups  of  readings,  {1(A),  2(B)},  {2(B),  1(A)},  {2(A),  1(B)} 
and  {1(B),  2(A)}.  A  balanced  cross-over  design  would  assign  radiologists  randomly 
to  these  four  reading  assignments.  The  difference  in  the  sensitivity  and  specifidty 
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between  modality  A  and  B  can  be  calculated  by  simply  pooling  aU  relevant  readings 
for  modality  A  and  similarly  for  modality  2.  Inference  for  the  difference  foUows  in 
the  same  fashion  as  that  described  for  the  change  induced  by  intervention  m  the 
intervention  group  of  our  study  but  that  now  there  are  4  rather  than  2  strata  of 
radiologists  defined  by  the  image  reading  set  assignments. 

Power  calculations  for  reading  studies  are  not  straightforward  due  in  part  to 
correlations  induced  by  images  and  readers.  That  is,  for  each  image  there  are  multiple 
readings.  Moreover,  each  reader  provides  multiple  readings  and  radiologist  specific 
sensitivities  and  specificities  are  correlated.  We  propose  valid  and  simple  analyses 
for  dealing  with  these  factors  but  power  calculations  required  a  computer  simulation 
approach.  We  found  the  process  of  developing  the  computer  simulation  study  to  be 
a  useful  exercise.  It  compels  one  to  think  through  the  processes  generating  study 
data.  It  also  allows  one  to  experiment  with  the  assumptions  and  design  easily.  For 
example,  we  considered  designs  which  included  a  larger  number  of  film  sets  to  be 
read  in  the  study  and  found  that  the  study  power  was  decreased  slightly  due  to  the 
extra  variation  introduced.  Computer  simulations  also  allow  one  to  check  how  test 
statistics  perform  under  the  null  hypothesis  with  sample  sizes  proposed  in  the  study. 
Hence  one  can  check  if  inference  based  on  large  sample  theory  is  valid  in  the  setting 
where  it  is  to  be  applied.  We  suggest  that  simulation  studies  are  a  useful  approach  to 
power  calculations  in  any  setting,  though  given  the  complexities  in  radiology  reading 
studies,  the  case  for  the  technique  in  this  setting  is  particularly  strong. 
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appendix 


1.  Variance  Estimators  for  Change  in  Overall  Sensitivity  and  SpeciBcity. 

The  diange  in  the  overall  sensitivity  defined  m  Section  4  can  be  written  formally 
mathematically  as 


Arfsensitivity)  =  —  \  (^r.post 

^r:(ordcr=l,2) 


,p«)  +  Z)  (^r.post  -  Srjp 
r:(ord€r=:2,l) 


where  Sr.^  is  the  observed  sensitivity  for  radiologist  r  with  his  pre-intervention  fUm 
set  and  5.^  is  the  corresponding  quantity  post-intervention.  Observe  that  the  order 
of  film  sets  essentially  defines  two  strata  in  this  setting  and  the  notation  (order  =  1, 2) 
(or  (order  =  2,1))  used  to  denote  the  stratum  in  the  summation  indicates  that  it 
includes  only  radiologists  assigned  sets  in  the  order  set  1  first  and  set  2  second  (or 
set  2  first  and  set  1  second).  The  variance  of  AT(sensitivity)  can  be  estimated  using 
the  variance  of  a  stratified  sample  mean  V  =  (V(i.2)  +  ^(2,i))/^Tj  where  V(i.2)  is  the 
sample  variance  of  the  quantities  (5.,p,.  -  5,.p«t)  in  the  stratum  (order=l,2),  and  V2 
is  the  analogous  quantity  in  the  other  stratum.  The  ratio  AT(sensitivity)/>/^^  can  be 
compared  with  a  standard  normal  distribution  to  test  for  a  change  in  the  sensitivity 


whidi  is  statistically  significantly  different  from  0. 


2.  Chi-squared  Test  Statistics  for  Bivariate  Analyses. 

To  simultaneously  test  the  null  hypotheses  that  both  the  sensitivity  and  speafiaty 
are  unchanged  in  the  intervention  group,  Eo  ’  Aj (sensitivity)  =  0  =  Ar  (Specificity), 
the  following  test  statistic  can  be  used 

,  * -1  Ar(sensitivity) 

[Ar(sensitivity)  AT(specifidty)]  AT(specificity) 
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where  the  square  bracket  notation  is  used  to  denote  vectors  and  is  the  inverse 
of  a  square  matrix  This  matrix  is  a  vanance-covariance  matrix  for  the 
two-dimensional  statistic  [Ax(sensitivity)  Aj*(specificity)],  and  is  the  analogue  of  the 
variance  V  defined  above  in  relation  to  the  one-dimensional  quantity  Aj (sensitivity). 
Formally  we  write 

Er  =  +  ii”’} 

where  is  the  sample  variance-covariance  matrix  for  the  quantities 

{Sr^post  -  ^r,post  "  m  the  stratum  (order  =  1,2),  and  Er'  is  the  analogous 

quantity  calculated  for  the  other  stratum.  The  test  statistic  is  compared  with  a 
standard  cW-squared  distribution  with  2  degrees  of  freedom  in  order  to  test  the  null 
hypothesis  concerning  changes  in  sensitivities  and  specificities. 

Consider  now  the  component  of  the  data  analysis  concerning  the  comparison  of 
changes  between  intervention  and  control  groups.  Using  a  subscript  C  to  denote  the 
control  group  in  analogy  with  our  use  of  the  subscript  T  to  denote  the  intervention 
group,  we  define  the  statistics  Ac'(sensitivity),  Ac'(spccificity)  and  Ec**  The  esti¬ 
mated  differences  between  the  groups  in  changes  of  sensitivities  and  specificities  can 
be  written  as  A2'(sensitivity)  —  A<7(sensitivity)  and  A7’(specificity)  —  Ac7(specificity), 
respectively.  The  hypothesis  that  the  changes  are  the  same  for  intervention  and 
control  groups  can  be  tested  by  comparing  the  statistic 

1-1  Ar(sens)  —  Ac(sens) 

[AT(sens)  -  Ac(sens)  Ar(spec)  -  Ac(spec)J  [Er  +  EcJ  ]Ar(spec)  -  Ac(spec) 

with,  the  quantiles  of  a  chi-squared  distribution  with  2  degrees  of  freedom,  where  we 
use  the  abbreviations  ‘sens’  and  ‘spec’  to  denote  ‘sensitivity’  and  ‘spedficity  in  the 
above  expressions. 
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3.  Mixed  Models  for  Reading  Accuracies. 

Section  5  outlines  a  statistical  model  for  sensitivity  and  specificity  parameters 
which  vary  with  reader  and  image.  Here  we  present  a  more  formal  and  precise 
definition  of  this  model.  For  radiologist  r  on  diseased  film  i,  we  write  the  chance  of 
correctly  identifying  it  as  diseased  pre-intervention  using  a  logistic  model  as 

=  exp{fi^  d-  7f  +  fir)/  (l  +  exp{/f^  +  7^  + 

where  7®  and  are  random  variables  spedfic  to  this  film  and  radiologist,  re¬ 
spectively.  For  the  average  radiologist  =  0,  and  for  the  average  film  7;  = 
0.  Thus  for  the  average  radiologist  on  the  average  film  the  sensitivity  is  = 
exp{/x^}/(l  -1-  e\p{n^}).  The  films  vary  in  difficulty  in  the  sense  that  the  average 
radiologist  has  a  lower  sensitivity  on  some  films  and  a  higher  sensitivity  on  oth¬ 
ers.  Mathematically  this  translates  into  allowing  7f  to  vary.  We  choose  it  as  a 
random  variable  so  that  the  average  radiologist’s  sensitivity  to  different  films  varies 
uniformly  in  an  interval  (5^  —  -f  o,^).  Technically  this  is  achieved  by  letting 

7P  =  ]ii{UP/{l  -  UP)}  -  pP  where  IjP  is  a  random  variable  with  a  uniform  distri¬ 
bution  in  (5®  -  aP,  -f  P).  The  radiologists  also  vary  amongst  themselves  in  their 
sensitivities  to  the  same  film  and  this  inter-rater  variation  translates  into  allowing 
to  vary.  We  simulated  data  so  that  on  the  avCTage  diseased  film  (i.e.,  —  0)  the 

sensitivities  of  radiologists  varied  uniformly  in  {S^  —  iP ■,  -|-  P)-  Again,  technically 

we  let  =  ln{C/f /(I  -U^)}-  pP  where  is  a  random  variable  with  a  uniform 

distribution  on  the  interval  {S^  —  iP ■,  -b  lP)~ 

Turning  now  to  specificities,  we  write  the  specificity  for  radiologist  r  on  non- 
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diseased  film  j  pre-intervention  as 


PjJ-  =  expiii^  +  'tf  +  /??} /(I  +  exp{/f^  +  7f  +  ^r)) 

where  in  analogy  with  the  above  notation  for  diseased  films,  the  average  radiologist  on 
the  average  film  has  spedficity  5^  =  exp{//^}/(H-exp{/x^)  and  parameters  and 
6^  indicate  variation  in  the  specificity  with  film  and  radiologist.  As  argued  in  section 
5,  data  should  be  generated  so  that  the  and  are  negatively  correlated.  We 
incorporated  this  into  the  simulation  by  first  generating  the  sensitivity  radiologist- 
specific  random  effect  parameter,  (i.e.,  his/her  sensitivity  to  the  average  film) 
which  is  based  on  the  random  variable  i/jP,  and  then  letting  the  corresponding  random 
variable  for  the  specificity  random  effect  be  defined  as 

Thus  if  the  radiologist’s  sensitivity  is  xxbP  above  the  average  radiologist’s  sensitivity 
to  the  average  film,  5®,  his/her  specificity  will  be  x  x  6®  below  the  average  specificity 
to  the  average  film. 

Our  model  postulates  that  after  intervention  the  quantities  5®  and  are  changed 
to  new  values  but  that  the  radiologist  and  image-specific  parameters  remain  un¬ 
changed.  Thus,  suppose  that  after  intervention  the  sensitivity  of  the  average  radiol¬ 
ogist  to  the  average  film  is  exp(^^  +  or®}/(l  -f  exp{fi^  +  o^}).  Then  the  chances 
that  radiologist  r  will  correctly  classify  film  t  pre-  and  post-intervention  are 

^r?i.pr«  =  exp{/X®  +  7f  +  }/(l  +  €Xp{M®  +  7f  +  fir)) 

and 

■P°jx>«  =  +  7?  +  ^}/(l  +  +  <*“  +  'li’  +  }). 
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respectively.  Similarly  the  postulated  change  in  specifies  a  parameter  oP  (anal¬ 
ogous  to  oP^  which  facilitates  calculation  of  post-intervention  specificities.  Having 
chosen  values  for  the  various  parameters  (^fi^,oP ^aP ^iP)  and  {^yP ^oP ^a.~ ^hP),  this 
completes  the  first  step  of  the  simulation  power  calculation  method,  namely  spedfi- 
cation  of  accuracy  parameter  distributions  pre-intervention  and  intervention  effects. 


O0044 


Table  1.  Study  power  to  detect  a  10%  increase  in  sensitivity  with  no 
accompanying  effect  on  specificity  pre-  versus  post-intervention  within  the 
intervention  group  and  power  to  detect  a  10%  increase  in  sensitivity  in  the 
intervention  group  versus  no  change  in  the  sensitivity  in  the  control  group  when 
specificites  in  both  groups  remain  unchanged  after  the  Intervention.  All  tests  are 
two  sided  and  are  tested  at  a  significance  level  of  .05. 


Readers 
Per  Group  (/Jy) 
20 
20 
20 
20 
20 
20 
20 
20 

30 
30 
30 
30 
30 
30 
30 
30 

40 
40 
40 
40 
40 
40 
40 
40 


Films 
Per  Set  (jlf ) 
30 
30 
30 
30 
45 
45 
45 
45 

30 
30 
30 
30 
45 
45 
45 
45 

30 
30 
30 
30 
45 
45 
45 
45 


Power 


Pre-intervention  Pre-intervention  Within  Comparison  with 

jensitivity  Specificity  Intervention  Group  Control  Group 


.70 

.70 

.80 

.80 

.70 

.70 

.80 

.80 

.70 

.70 

.80 

.80 

.70 

.70 

.80 

.80 

.70 

.70 

.80 

.80 

.70 

.70 

.80 

.80 


.70 

.80 

.70 

.80 

.70 

.80 

.70 

.80 

.70 

.80 

.70 

.80 

.70 

.80 

.70 

.80 

.70 

.80 

.70 

.80 

.70 

.80 

.70 

.80 


.70 

.66 

.79 

.77 

.81 

.82 

.91 

.92 

.81 

.83 

.93 

.91 

.94 

.95 

.99 

.99 

.92 

.94 

.97 

.98 

.98 

.99 

.99 

.99 


.38 

.34 

.45 

.44 

.48 

.53 

.61 

.64 

.48 

.52 

.60 

.61 

.66 

.66 

.80 

.79 

.61 

.60 

.73 

.75 

.79 

.80 

.88 

.89 


Table  2*  Study  Power  to  detect  various  configurations  of  changes  in  the 
intervention  group  using  a  study  design  with  30  readers  and  45  fi^s  per  set.  The 
pre-intervention  specifidty  is  assumed  to  be  .70  in  all  cases.  The  intervention 
induced  change  in  sensitivity  is  denoted  Ay  (sens)  and  in  specificity  is  denoted 
Ax(spec). 


Pre- Intervention 


Sensitivity 

Ar(sens) 

Ay(spec) 

Power 

.60 

+0.10 

0.00 

0.90 

.70 

+0.10 

0.00 

0.95 

.80 

+0.10 

0.00 

0.98 

.60 

+0.05 

0.00 

0.35 

.70 

+0.05 

0.00 

0.39 

.80 

+0.05 

0.00 

0.50 

.60 

+0.05 

+0.05 

0.66 

.70 

+0.05 

+0.05 

0.68 

.80 

+0.05 

+0.05 

0.71 

True  positive  rate 


False  positive  rate 


Figure  1 :  An  hypothetical  setting  where  the  sensitivity  and  specifi^  associated 
with  the  clinically  relevant  criteria  are  unchanged  but  the  empirical  ROC 
curves  Indicate  a  benefit  of  intervention.  The  (false  positive,  true  positive) 
points  assodated  with  categories  5, 4, 3,  and  2  are  (.10,  .30),  (.25,  .70), 

(.45,  .85),  and  (.75,  .95)  respectively  pre-intervention  and  (.10,  .60),  (25, 70), 

(.45*,  ,85)!  and  (.55,  .95)  respectively  post-intervention. 

903/96 


n  r\  f\  /t  *y 


True  positive  rate 


Rgure  2:  An  hypothetical  setting  where  ROC  curve  is  unchanged  by  the  intervention 
but  there  is  a  clinically  relevant  benefit.  The  sensitivity  associated  with  the 
clinically  relevant  criterion  is  improved  from  .50  to  .70  while  the  assodated  fal^ 
positive  rate  remains  unchanged  at  0.09.  The  (false  positive,  true  positive)  points 
assodated  with  categories  5, 4, 3,  and  2  are  (.03,  .27),  (.09,  .50),  (.15,  .83),  and 
(.39,  .93)  pre-intervention  and  (.03,  .27),  (.09,  .70),  (.15,  .83),  and  (.39,  .93) 
post-intervention.  These  points  before  intervention  are  labelled  with  drdes  and 
after  intervention  are  labelled  with  triangles. 
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Figure  3:  Joint  confidence  regions  for  sensitivity  and  specifidty  both  pre  and  post  intervention 
aiong  with  a  joint  confidence  region  for  the  changes  in  these  parameters.  Data  used  in 
this  illustration  were  generated  using  computer  simulation  methods  described  in  sections 
5  and  6.  Points  correspond  to  observed  data  for  individual  radiologists. 
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I  Facility  Recruitment 


A.  Overview 

A  minimum  of  sixteen  rural  mammography  facilities  or  facility  groups  that  when 
combined  will  have  at  least  60  radiologists  reading  mammograms  for  them,  will  be 
recruited  to  participate  in  the  MQIP.  The  focus  of  the  recruitment  effort  will  be  on 
smaller,  more  rural  facilities  or  facility  groups  that  do  not  have  dedicated 
mammographers,  e.g.,  radiologists  who  read  only  mammograms.  A  facility  group  is 
defined  as  a  combination  of  facilities  that 

•  have  a  business  association;  or 

•  may  share  radiologists. 

Facility-associated  radiologists  may  read  for  a  second,  unrelated  facility  outside  of  the 
facility  group,  however,  the  second  clinic  may  not  be  part  of  another  MTR/MQIP 
selected  facility  group. 

After  selection  of  the  sixteen  (or  more)  facilities  targeted  for  the  MQIP  intervention,  each 
facility  will  first  be  recruited  to  participation  in  the  Mammography  Tumor  Registry  (MTR), 
a  state-wide  effort  to  link  facility  mammograms  with  the  state  cancer  registry.  Each 
facility  will  be  required  to  sign  a  Data  Release/Confidentiality  Agreement  that  allows  the 
MTR  to  link  mammograms  performed  at  the  facility  with  information  in  the  Washington 
State  Cancer  Registry. 

All  radiologists  reading  for  the  sixteen  targeted  facilities  will  be  notified  after  the  facility 
signs  the  agreement.  See  Section  II,  Radiologist  Recruitment  for  more  details. 

B.  Recruitment  Procedures 

All  targeted  facilities  will  be  recruited  to  the  MTR  in  the  same  manner. 

1 .  The  first  contact,  a  letter  addressed  to  the  facility/clinic  manager,  will  consist  of  the 
following: 

•  an  introduction  to  the  Mammography  Tumor  Registry  (Appendix  -  Letter  1); 

•  an  MTR  brochure  outlining  the  benefits  of  participation  in  the  registry 
(Appendix  -  promotional  1); 

•  a  return  card  (Appendix  -  Form  1)  on  which  the  facility  indicates  the  method 
-  paper  or  computer  -  used  to  collect  and  store  patient  information,  health 
history,  and  examination  information. 

2.  Approximately  ten  days  after  the  initial  mailing  to  the  facility,  a  field  coordinator  will 
telephone  to  confirm  the  information  on  the  returned  card  or  to  obtain  the  information 
if  the  card  was  not  returned.  In  addition  to  confirming  the  information  on  the 
returned  card,  calls  to  facilities  will  also  determine  the  following: 

•  the  correct  person  to  receive  future  MTR/MQIP  mailings  (sample  scripts. 
Appendix  -Forms  2  and  3); 

•  additional  mammography  facility  sites  associated  with  the  one  being 
contacted; 
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•  for  multiple  facilities  under  one  management,  information  about  whether  their 
database  is  shared  or  if  individual  databases  are  maintained  at  each  site; 

•  a  correct  business  office  address  and  telephone  number. 

3.  After  the  MTR  receives  the  data  collection  methods'  card  (or  after  the  facility  is 
contacted  by  phone  and  the  information  is  obtained),  the  appropriate  information 
packet  will  be  assembled  and  mailed.  The  information,  presented  in  a  folder, 
includes  the  following: 

•  a  cover  letter  to  the  facility  manager  or  designated  facility  personnel  that 
previews  the  MQIP  and  the  letter  that  will  go  to  the  facility’s  reading 
radiologists  offering  them  the  opportunity  to  be  part  of  the  MQIP  and  the 
randomized  CQI  study.  (Appendix  -  Letter  2a,  2b); 

•  an  MTR  brochure  (Appendix  -  Promotional  1); 

•  two  copies  of  the  Data  Release/Confidentiality  Agreement  -  one  to  be  kept 
for  the  facility  files  (Appendix  -  Form  4); 

•  a  facility  staff  directory  and  a  radiologist  directory  for  completion,  correction 
and/or  updating  (sample,  Appendix  -  Forms  5  and  6); 

•  questions  and  answers  about  the  MTR,  either  computer  or  paper  data 
collection -specific  (Appendix  -  Forms  7a  and  7b) 

•  facilities  that  collect  and  store  data  on  computer  will  receive  a  sample  of  the 
validation  report  (sample.  Appendix  -  Form  8) 

•  for  facilities  that  collect  and  store  information  on  paper  only,  a  draft  copy  of  a 
new  data  collection  form  that  they  are  urged  to  use  because  it  requests 
information  needed  for  linking  (Appendix  -  Form  9);  computerized  facilities 
receive  a  copy  to  confirmed  the  desired  data  to  be  collected; 

•  further,  paper-based  facilities  have  the  opportunity  to  personalize  the  paper 
data  collection  form  with  their  facility  name  by  returning  a  card  indicating  the 
heading  they  would  like  included  on  their  forms  (Appendix  -  Form  10); 

•  A  stamped  return  envelope. 

4.  If  there  is  no  response  from  the  facility,  e.g.,  no  return  of  the  documents,  within  ten 
working  days,  a  field  coordinator  will  telephone  to  answer  questions  and  check  on 
the  status  of  the  paperwork.  At  this  time,  the  field  coordinator  will  make  certain  the 
packet  arrived  and  offer  to  send  a  new  one  if  necessary.  Any  questions  about  MTR 
that  cannot  be  answered  by  the  field  coordinators  will  be  directed  to  the  appropriate 
MTR  staff. 

5.  After  receipt  of  the  signed  Data  Confidentiality  Form,  the  facility  is  considered  part  of 
the  MTR.  The  appropriate  MTR  data  staff  will  begin  working  with  the  facility  to 
extract  data  from  their  files  so  that  the  linking  process  with  the  Washington  State 
Cancer  Registry  data  can  begin. 
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II  Radiologist  Recruitment 

A.  Overview 

All  radiologists  reading  for  the  targeted  participating  facilities  or  facility  groups  will  be  the 
focus  of  recruitment  to  the  MQIP.  After  the  facility  returns  the  corrected  radiologist  and 
staff  directories,  the  information  in  the  MTR  FACTS  system  is  updated  and  the 
radiologists’  recruitment  begins. 

At  that  time  the  radiologists  will  be  informed  about  the  MQIP  and  urged  to  sign  the 
participation  agreement  that  allows  potential  randomization  to  the  CQI  program.  Later 
all  radiologists  will  be  invited  to  submit  interesting  cases  for  inclusion  in  the  CQI.  Finally, 
participating  radiologists  will  be  offered  the  opportunity  to  sign  the  Radiologist’s  Data 
Release  form.  A  radiologist’s  signature  on  the  form  allows  him  to  receive  regular, 
patient-identified  reports  on  the  mammograms  he  reads.  When  retrospective  data 
becomes  available,  the  radiologist  will  receive  information  that  identifies  patients  for 
whom  he  read  a  mammogram  who  were  later  diagnosed  with  breast  cancer. 

B.  Recruitment  Procedures 

All  radiologists  reading  for  the  MTR  targeted  facilities  will  be  recruited  in  the  same 
manner. 

1 .  An  initial  letter  to  be  sent  to  each  targeted  radiologist  will  include  the  following 
information: 

•  an  introduction  to  the  MQIP  with  emphasis  on  their  associated  facility’s 
interest  in  participation  (Appendix  -  Letter  3); 

•  the  MQIP  Consent  to  Participate  Agreement  which  details  the  project  and  the 
randomization  process  (Appendix  -  Form  11); 

•  A  stamped,  return  envelope. 

2.  Whether  or  not  the  signed  consent  is  returned,  each  targeted  radiologist  will  receive 
a  second  mailing  ten  days  after  the  first.  This  mailing  will  contain  the  following: 

•  a  cover  letter  which  reminds  the  radiologist  to  sign  and  return  the 
Participation  Agreement  if  it  hasn’t  been  returned  yet; 

•  a  request  for  interesting  mammography  films  to  be  considered  for  possible 
inclusion  in  the  CQI  software  program  (Appendix  -  Letter  4); 

•  four  copies  of  the  mammography  Case  Selection  forms  (Appendix  -  Form 
12); 

•  a  stamped  return  envelope. 

3.  If  there  is  no  response,  e.g.,  the  signed  Participation  Agreement  is  not  received  ten 
working  days  after  the  second  mailing,  the  field  coordinators  and/or  the  project  co¬ 
investigator  will  telephone  the  radiologist  to 

•  offer  to  send  the  form  again, 

•  offer  to  answer  questions  about  the  MQIP  or  direct  questions  to  the 
appropriate  staff  person,  and 

•  urge  the  radiologist  to  submit  interesting  mammography  films  for  possible 
inclusion  in  the  CQI. 
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The  radiologist  will  receive  reminder  calls,  or  a  second  mailing,  until  the  signed 
agreement  is  returned  or  the  radiologist  declines  further  participation. 

4.  After  all  targeted  radiologists  have  either  returned  the  signed  participation 
agreement  (or  declined  participation),  randomization  to  the  CQI  will  take  place. 
Radiologists  (N  =  >30)  at  one-half  of  the  participating  facilities  will  be  randomized  to 
take  part  in  the  Continuous  Quality  Improvement  (CQI)  program.  After 
randomization,  all  radiologists  will  be  notified  about  their  status  in  the  CQI  program 
and  their  opportunity  to  receive  the  Radiologist’s  Quality  Improvement  Report. 

5.  The  notification  letter  for  radiologists  randomized  to  the  CQI  will  include; 

•  a  thank  you  for  participation,  information  about  the  randomization,  an 
introduction  to  the  MTR  emphasizing  the  benefits  for  radiologists,  and  a 
preview  of  the  first  CQI  scheduling  phone  call  (Appendix  -  Letter  5a); 

•  an  MTR  brochure  (Appendix  -  Promotional  1); 

•  a  Radiologist  Data  Release  form  (Appendix  -  Form  13); 

•  an  MQIP  timeline  -  in  development  (Appendix  -  Form  14); 

•  a  stamped  addressed  envelope. 

6.  For  radiologists  not  randomized  to  the  CQI,  the  letter  will  include; 

•  a  thank  you  for  participation  and  an  introduction  to  the  MTR  emphasizing  the 
benefits  for  radiologists  (Appendix  -  Letter  5b): 

•  a  Radiologist’s  Data  Release  form  (Appendix  -  Form  13); 

•  a  stamped  addressed  envelope. 

The  radiologist's  signature  on  the  data  release  form  allows  the  radiologist  to  begin 
receiving  the  Radiologist’s  Quality  Improvement  report  that  identifies  patients  for  whom 
he/she  read  a  mammogram  and  who  were  later  diagnosed  with  breast  cancer.  These 
reports  will  be  sent  when  there  is  sufficient  data  in  the  data  base  to  link  the  patients 
through  the  state  cancer  registry.  While  it  is  desirable  that  the  radiologist  agree  to 
receive  the  confidential  report,  a  radiologist  does  not  have  to  sign  the  data  release  form 
in  order  to  participate  in  the  CQI. 
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The  Continuous  Quality  Improvement  Sessions 


A.  Overview 

The  CQI  sessions  are  designed  to  determine  the  following: 

1 .  determine  inter-rater  variability  in  film  interpretation  before  and  after  the 
implementation  of  the  program  which  tests  the  radiologists  on  a  set  of 
mammograms  selected  specifically  for  their  teaching  value; 

2.  determine  post-CQi  intra-rater  variability  in  film  interpretation; 

3.  determine  if  digitized  mammograms  can  be  interpreted  with  the  same 
accuracy  as  high-quality  copies  of  mammograms; 

4.  determine  if  the  accuracy  of  film  interpretation  depends  on  covariates  such 
as  the  woman’s  age  and  breast  density. 

B.  Procedures 

The  CQI  sessions  are  intended  to  occur  over  a  12-14  month  period  with  every  effort 
made  to  have  a  consistent  and  approximately  equal  amount  of  time  (8  -12  weeks) 
between  sessions. 

1.  To  schedule  the  first  session,  the  field  coordinators  will  phone  each  physician’s 
office  and  schedule  either  with  the  physicians  individually  or  with  the  office  manager. 
Every  effort  will  be  made  to  schedule  all  physicians  in  a  practice  during  the  same 
field  coordinator  visit.  Office  staff  will  be  informed  of  the  necessary  requirements  for 
the  sessions  including 

•  A  separate  room; 

•  space  to  set  up  the  light  box; 

•  space  for  the  radiologist  to  sit  to  use  the  laptop  computer; 

•  two  electrical  outlets. 

The  CQI  sessions  will  involve  the  radiologist  reading  45  (sessions  1  and  4)  or  30 
(sessions  2  and  3)  high  quality  copies  of  mammography  films.  He/she  will  enter  an 
assessment  of  each  case  on  a  digitized  image  of  the  film  on  a  laptop  computer.  A  field 
coordinator  will  facilitate  each  session  by 

•  introducing  the  session; 

•  giving  basic  instructions; 

•  positioning  the  films  on  the  light  box; 

•  giving  instructions  for  the  CQI  computer  program;  and 

•  ensuring  that  the  sessions  follow  project  protocol. 

2.  Before  the  first  session,  the  physician  will  receive: 

•  a  reminder  postcard  indicating  the  day/date/time  of  the  session  to  be  mailed 
about  one  week  in  advance  (Appendix  -  Form  15); 

•  a  reminder  phone  call  two  days  before  the  session. 

3.  If  a  physician  has  not  signed  the  radiologist  data  release  form,  he  will  be  offered  the 
opportunity  to  do  so  at  the  first  CQI  session.  Field  coordinators  will  carry  the  forms 
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with  them  and  offer  to  answer,  or  direct  to  any  MTR  staff  person,  questions  about 
the  radiologist  report. 

4.  At  each  session  beginning  with  the  first,  the  following  session  will  be  scheduled.  The 
field  coordinator  will  remind  the  physician  about  the  timeline,  which  is  approximately 
8-10  weeks  between  sessions,  and  schedule  the  next  session  on  her  calendar. 
Follow-up  will  include 

•  A  reminder  postcard  mailed  two  weeks  prior  to  each  of  sessions  2-5  which 
allows  the  physician  time  to  reschedule  if  necessary; 

•  a  phone  call  will  be  made  a  day  or  two  before  the  session  as  a  courtesy 
reminder. 

6.  When  courtesy  is  extended  to  field  coordinators  by  the  radiologists’  staff  or  facility 
personnel,  individual  handwritten  thank  you  notes  will  be  sent.  All  correspondence 
will  be  entered  into  the  FACTS  system. 

After  all  five  CQI  sessions  are  completed,  the  radiologist  will  receive  certification  notice 

of  the  10  CME  credits  earned. 
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IV  Appendix 


A.  Letters 

B.  Forms 

C.  Promotional  Materials 
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September  18,  1997 


Letter  1 

^^tial  Facility  Contact 


«cfacnm» 

Attn:  «ccontact» 

«caddrl» 

«caddr2» 

«ccity»,  «cstate»  «czip» 

Dear  «ccontact»: 

In  a  survey  of  mammography  facilities  that  was  conducted  last  fall,  your  facility  expressed 
interest  in  the  Washington  State  Mammography  Tumor  Registry  (MTR).  The  MTR  provides 
follow-up  mammography  information  to  facilities  and  offers  researchers  the  opportunity  to  gain 
insight  into  issues  related  to  breast  cancer  detection.  In  the  enclosed  brochure  that  summarizes 
the  benefits  of  the  MTR,  you  will  note  that  a  key  benefit  of  participation  is  a  report  containing 
information  that  helps  satisfy  the  audit  recommendations  of  the  Mammography  Quality 
Improvement  Act  (MQSA). 

Please  complete  and  return  the  enclosed  card  indicating  how  your  facility  collects  and  stores 
patient  and  examination  information.  We  will  then  send  you  an  information  packet  and  details 
about  participation  in  the  MTR. 

As  an  MTR  participant,  your  facility  is  also  eligible  for  the  Mammography  Quality  Improvement 
Project  (MQIP).  The  MTR  and  the  Mammography  Quality  Improvement  Project  are  being 
conducted  by  investigators  at  the  Fred  Hutchinson  Cancer  Research  Center.  The  MQIP  is  aimed 
at  predominantly  rural  areas  of  Washington  State  where  mammography  use  is  being  studied.  In 
our  ongoing  effort  to  reduce  breast  cancer  mortality,  the  Hutchinson  Center  was  awarded  a  grant 
to  examine  methods  for  improving  the  quality  of  mammography. 

The  MQIP  will  offer  a  Continuing  Medical  Education  conference  for  facility  technologists  at  no 
cost  to  the  facility  or  the  technologist.  Each  radiologist  who  reads  mammograms  for  your 
facility  will  also  be  contacted  about  the  MQIP.  For  radiologists,  a  key  component  of  the  MQIP 
is  participation  in  a  Continuous  Quality  Improvement  (CQI)  program.  This  CQI  program  is 
designed  to  strengthen  film-reading  ability  by  providing  immediate  feedback  on  the  radiologist’s 
interpretation  of  a  set  of  mammograms  specifically  selected  for  their  teaching  value.  One-half  of 
the  participating  radiologists  will  be  randomly  selected  to  take  part  in  the  CQI  film-reading 
study.  Initially,  the  CQI  program  will  be  available  only  in  the  context  of  this  research  study. 
Radiologists  who  are  selected  for  the  film  reading  study  will  receive  Continuing  Medical 
Education  (CME)  credits  for  their  participation. 

We  look  forward  to  your  collaboration  with  us  on  the  Mammography  Tumor  Registry  and  the 
Mammography  Qualify  Improvement  Project.  If  you  have  any  further  questions,  please  contact 
the  project  office  at  (800)  224-233 1  or  (206)  667-6560. 

Sincerely, 


Nicole  Urban  ScD 
Principal  Investigator 


Enclosure: 


Brochure 

Card 

Stamped  return  envelope 
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September  18,  1997 


«cfacnm» 

Attn:  «ccontact» 

«caddrl» 

«caddr2» 

«ccity»,  «cstate»  «czip» 

Dear  «ccontact»: 

We  invite  your  facility  to  participate  in  the  Washington  State  Mammography  Tumor  Registry 
(MTR).  The  MTR  provides  follow-up  mammography  information  to  facilities  and  offers 
researchers  the  opportunity  to  gain  insight  into  issues  related  to  breast  cancer  detection.  In  the 
enclosed  brochure  that  summarizes  the  benefits  of  the  MTR,  you  will  note  that  a  key  benefit  of 
participation  is  a  report  containing  information  that  helps  satisfy  the  audit  recommendations  of 
the  Mammography  Quality  Standards  Act  (MQSA). 

Please  complete  and  return  the  enclosed  card  to  indicate  how  your  facility  collects  and  stores 
patient  and  examination  information.  We  will  then  send  you  an  information  packet  and  details 
about  participation  in  the  MTR. 

As  an  MTR  participant,  your  facility  is  also  eligible  for  the  Mammography  Quality  Improvement 
Project  (MQIP).  The  MTR  and  the  Mammography  Quality  Improvement  Project  are  being 
conducted  by  investigators  at  the  Fred  Hutchinson  Cancer  Research  Center.  The  MQIP  is  aimed 
at  predominantly  rural  areas  of  Washington  State  where  mammography  use  is  being  studied.  In 
our  ongoing  effort  to  reduce  breast  cancer  mortality,  the  Hutchinson  Center  was  awarded  a  grant 
to  examine  methods  for  improving  the  quality  of  mammography. 

The  MQIP  will  offer  a  Continuing  Medical  Education  conference  for  facility  technologists  at  no 
cost  to  the  facility  or  the  technologist.  Each  radiologist  who  reads  mammograms  for  your 
facility  will  also  be  contacted  about  the  MQIP.  For  radiologists,  a  key  component  of  the  MQIP 
is  participation  in  a  Continuous  Quality  Improvement  (CQI)  program.  This  CQI  program  is 
designed  to  strengthen  film-reading  ability  by  providing  immediate  feedback  on  the  radiologist’s 
interpretation  of  a  set  of  mammograms  specifically  selected  for  their  teaching  value.  Gne-half 
of  the  participating  radiologists  will  be  randomly  selected  to  take  part  in  the  CQI  film-reading 
study.  Initially,  the  CQI  program  will  be  available  only  in  the  context  of  this  research  study. 
Radiologists  who  are  selected  for  the  film-reading  study  will  receive  Continuing  Medical 
Education  (CME)  credits  for  their  participation. 

We  look  forward  to  your  collaboration  with  us  on  the  Mammography  Tumor  Registry  and  the 
Mammography  Quality  Improvement  Project.  If  you  have  any  further  questions,  please  contact 
the  project  office  at  (800)  224-233 1  or  (206)  667-6560. 

Sincerely, 


Nicole  Urban,  ScD 
Principal  Investigator 

Enclosure:  Brochure 

Card 

Stamped  return  envelope 
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September  18,  1997 


Letter  2a 

Packet  Cover  Letter 
Oonputer-based  Facility 


«cfacnm» 

Attn:  «ccontact» 

«caddrl» 

«caddr2» 

«ccity»,  «cstate»  «czip» 

Dear  «ccontact»: 

Thank  you  for  your  interest  in  the  Mammography  Tumor  Registry  (MTR).  The  enclosed 
material  includes  a  complete  description  of  the  MTR  and  the  initial  paperwork  required  for 
enrollment.  After  reading  the  project  description,  we  ask  you  to  do  the  following: 

1 .  Sign  and  date  both  copies  of  the  Data  Release/Confidentiality  Agreements.  Keep  a  copy 
for  your  files  and  return  one  to  the  MTR.  Our  receipt  of  the  signed  agreement  confirms 
your  participation. 

2.  Review  and  complete  or  correct  the  Radiologist  Directory  and  return  the  form  to  us.  We 
need  current  information  because  each  radiologist  who  reads  mammograms  for  your 
facility  will  receive  a  letter  of  introduction  to  the  Mammography  Quality  Improvement 
Project  (MQIP).  The  introductoiy  letter  includes  the  Participation  Agreement  for  the 
radiologist’s  signature. 

3.  Review  and  complete  or  correct  the  Facility  Staff  Directory  so  that  our  contacts  are  with 
the  appropriate  personnel.  To  help  you  determine  which  staff  person  should  receive  and 
review  the  data  validation  reports,  we  include  a  sample  report. 

4.  In  order  to  link  mammography  results  with  cancer  incidence  data,  standardized  patient 
and  exam  information  are  required.  We  include  a  copy  of  the  patient  history  and 
medical  record  form  used  by  MTR  facilities  that  do  not  have  a  computerized  system  for 
data  entiy.  To  help  us  work  with  your  facility,  return  the  following  forms: 

a.  a  copy  of  your  patient  information  and  medical  history  form  with  the  data 
you  enter  highlighted  and 

b.  a  copy  of  the  exam  form  with  all  data  that  you  currently  enter  highlighted. 

If  you  have  any  questions  after  reading  the  information  in  this  packet,  please  call  1(800)  224- 
233 1  or  (206)  667-6560  to  speak  to  someone  on  our  staff. 

We  are  pleased  to  have  you  working  with  us  on  this  valuable  research  endeavor. 

Sincerely, 


Nicole  Urban,  ScD 
Principal  Investigator 
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Letter  2b 

Packet  Cover  Letter 
Paper-Based  Facility 

September  18, 1997 

«cfacnm» 

Attn:  «ccontact» 

«caddrl» 

«caddr2» 

«ccity»,  «cstate»  «czip» 

Dear  «ccontact»: 

Thank  you  for  your  interest  in  the  Mammography  Tumor  Registry  (MTR).  The  enclosed 
material  includes  a  complete  description  of  the  MTR  and  the  initial  paperwork  required  for 
enrollment.  After  reading  the  project  description,  we  ask  you  to  do  the  following: 

1 .  Sign  and  date  both  copies  of  the  Data  Release/Confidentiality  Agreements.  Keep  a  copy 
for  your  files  and  return  one  to  the  MTR.  Our  receipt  of  the  signed  agreement  confirms 
your  participation. 

2.  Review  and  complete  or  correct  the  Radiologist  Directory  and  return  the  form  to  us.  We 
need  current  information  because  each  radiologist  who  reads  mammograms  for  your 
facility  will  receive  letter  of  introduction  to  the  Mammography  Quality  Improvement 
Project  (MQIP).  The  introductory  letter  includes  the  Participation  Agreement  for  the 
radiologist’s  signature. 

3.  Review  and  make  necessary  changes  to  the  Facility  Staff  Directory  so  that  our  contacts 
are  with  the  appropriate  personnel.  Return  the  corrected  form. 

4.  To  facilitate  the  collection  of  uniform  data  from  all  state  mammography  facilities,  the 
MTR  developed  a  standardized  patient  history  and  medical  record  form  in  accordance 
with  the  April  1996  Proposed  Rules  for  the  MQSA,  Section  900.12.  A  copy  of  the  form 
is  enclosed.  The  MTR  will  print  these  personalized  forms  for  your  facility.  Return  the 
card  attached  to  the  form,  indicating  the  information  you  would  like  included  in  the  top 
margin.  For  example,  you  may  request  that  the  facility  name  or  you  may  request  white 
space  for  a  patient  identification  label.  The  forms  are  provided  in  a  carbon-less  format; 
the  facility  keeps  the  original  and  returns  a  copy  to  the  MTR  on  a  regular  basis. 

If  you  have  any  questions  after  reading  the  information  in  the  packet,  please  call  1(800)  224- 
233 1  or  (206)  667-6560  to  speak  to  someone  on  our  staff. 

We  are  pleased  to  have  you  working  with  us  on  this  valuable  research  endeavor. 

Sincerely, 


Nicole  Urban,  ScD 
Principal  Investigator 


onoei 
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September  18,  1997 

«cpersfiim»  «cperslnm»,  «ctitle» 

«caddrl» 

«caddr2» 

«ccity»,  «cstate»  «czip» 

Dear  Dr.  «cperslnm»: 

A  facility  for  which  you  regularly  read  mammograms,  «cgrpnm»,  has  expressed  interest  in 
participating  in  the  Mammography  Quality  Improvement  Project  (MQIP),  a  research  study 
conducted  by  investigators  at  the  Fred  Hutchinson  Cancer  Research  Center. 

The  Mammography  Quality  Improvement  Project  is  aimed  at  predominantly  rural  areas  of 
Washington  State  where  mammography  use  is  being  studied.  In  our  ongoing  effort  to  reduce 
breast  cancer  mortality,  the  Hutchinson  Center  was  awarded  a  grant  to  examine  methods  for 
improving  the  quality  of  mammography  screening. 

The  attached  participation  agreement  explains  all  aspects  of  the  MQIP.  One  component  of  the 
MQIP  is  a  research  project  that  tests  the  efficacy  of  a  Continuous  Quality  Improvement  (CQI) 
film-reading  program  for  radiologists.  One-half  of  the  participating  radiologists  will  be 
randomly  assigned  to  the  CQI  intervention  group.  In  brief,  the  intervention  consists  of  five 
two-hour  film-reading  sessions  conducted  over  a  12  -  14  month  period.  In  the  first  four  CQI 
sessions,  the  radiologist  reads  a  series  of  mammograms  (cases)  and  enters  an  assessment  of 
each  case  on  a  digitized  computer  image.  At  the  fifth  session,  the  radiologist  reads  and 
assesses  digitized  mammograms  only.  This  last  session  will  determine  the  feasibility  of 
disseminating  the  CQI  program  via  the  Internet. 

Films  are  read  using  standard  assessment  codes.  On  mammograms  coded  as  possible  cancer 
(3, 4  or  5),  the  radiologist  indicates  the  location  of  the  abnormality  on  the  digitized  image. 
Feedback  is  offered  at  the  conclusion  of  each  session  and  the  radiologist  is  given  the 
opportunity  to  review  individual  films.  Ten  hours  of  continuing  medical  education  (CME) 
credits  are  awarded  after  the  completion  of  the  five  sessions.  All  CME  paperwork  will  be 
handled  by  our  office. 

Please  read  the  attached  participation  agreement  carefully,  initial  each  page,  complete  and  sign 
the  final  page,  and  return  the  agreement  in  the  enclosed  envelope.  You  will  receive  a  copy  for 
your  files.  After  randomization  occurs,  you  will  be  notified  about  your  status.  If  you  are 
randomized  to  the  CQI  intervention  group,  you  will  be  contacted  to  schedule  the  first  session  in 
early  1998. 

If  you  have  any  questions  about  the  MQIP,  please  call  the  project  office  at  (800)  224-233 1  or 
(206)  667-6560.  A  staff  person  will  be  happy  to  help  you. 

Thank  you  for  the  prompt  return  of  the  enclosed  paperwork. 

Sincerely, 


Letter  3 

Radiologist's  Initial  Letter 
Introduction  to  MQIP 


Nicole  Urban,  ScD 
Principal  Investigator 

0  n  r  S2 

Enclosure:  MQIP  Consent  to  Participate 

Stamped  Return  Envelope 
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September  18,  1997 


Letter  4 

Radiologist's  Reminder 
Film  Solicitation 


«cpersfnm»  «cperslnm»,  «ctitle» 

«caddrl» 

«caddr2» 

«ccity»,  «cstate»  «czip» 

Dear  Dr.  «cperslnm»: 

Recently  you  received  a  letter  inviting  you  to  participate  in  the  Mammography  Quality 
Improvement  Project  (MQIP).  If  you  have  not  returned  the  participation  agreement,  please 
take  a  minute  or  two  to  sign  and  return  it.  We  will  mail  you  a  signed  and  dated  copy  of  the 
agreement. 

You  will  recall  that  one  aspect  of  the  MQIP  is  the  Continuous  Quality  Improvement  (CQI) 
film-reading  program.  The  radiologists  who  are  randomly  assigned  to  the  CQI  program  will 
read  180  mammography  films  at  five  film  reading  sessions  over  a  12  -  14  month  period. 

The  films  used  in  the  CQI  sessions  are  being  solicited  from  radiologists  around  the  state.  We 
hope  you  will  submit  4-6  interesting  cases  for  the  pool  of  mammography  films  to  be  reviewed 
for  study  inclusion.  We  are  looking  for  an  equal  number  of  benign  and  malignant  assessments 
with  about  25%  from  women  under  the  age  of  50. 

Consider  the  following  criteria  when  you  review  cases  for  submission: 

•  The  woman  was  asymptomatic  at  the  time  of  the  mammogram. 

•  The  interpretation  was  challenging. 

•  Malignant  assessments  have  biopsy  correlation. 

•  Benign  assessment  is  determined  either  by  1)  a  woman’s  cancer  free  status  two  years  after 
the  mammogram;  2)  additional  evaluation  resulting  in  a  benign  finding;  or  3)  a  negative 
biopsy  result. 

Please  complete  one  of  the  enclosed  case  selection  forms  for  each  case  you  submit  and  return 
them  by  insert  date.  Do  not  send  original  mammograms.  Our  staff  will  contact  the  clinic 
storing  the  films  and  make  the  necessary  arrangements  to  obtain  them.  All  information  you 
submit  remains  strictly  confidential.  Before  the  review  and  selection  process,  patient,  clinic, 
and  physician  identifiers  are  masked  on  the  films.  Films  chosen  for  study  inclusion  are  masked 
during  the  copying  and  digitization  procedures. 

All  films  will  be  returned  to  the  clinic  within  eight  weeks.  If  there  is  an  emergency,  immediate 
return  can  be  arranged  by  calling  the  project  office  at  (800)  224-233 1  or  (206)  667-6560.  If 
you  have  any  questions  about  case  submission,  you  can  reach  a  project  staff  person  at  the  above 
numbers. 

Thank  you  for  taking  time  to  consider  films  for  submission. 

Sincerely, 


Nicole  Urban,  ScD 
Principal  Investigator 

Enclosure:  Case  Selection  Forms 

Stamped  return  envelope 
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September  18, 1997 


Letter  5a 

Notifi^tion  of  Eandaniza- 
tion  Status 
Receive  GQI 

ttvhr^ducHetc  mtit) 


«cpersfnm»  «cperslnm»,  «ctitle» 

«caddrl» 

«caddr2» 

«ccity»,  «cstate»  «czip» 

Dear  Dr.  «cperslnm»: 

Thank  you  for  participating  in  the  Mammography  Quality  Improvement  Project  (MQIP). 
You  have  been  randomized  to  the  Continuous  Quality  Improvement  (CQI)  film-reading 
intervention  group.  To  remind  you,  the  CQI  sessions  are  about  two  hours  in  length  and  will 
be  conducted  on-site  by  a  project  field  coordinator  over  a  12-14  month  period.  Please  refer 
to  the  enclosed  timeline  for  the  intervention  schedule.  A  field  coordinator  will  contact  you 
soon  to  schedule  the  first  session. 

You  may  also  request  information  from  the  Mammography  Tumor  Registry  (MTR).  The 
benefits  of  the  MTR  are  outlined  in  the  enclosed  brochure.  For  the  radiologist,  the  MTR  can 
provide  a  Radiologist  Quality  Improvement  Report.  These  reports  identify  women  for  whom 
you  read  a  mammogram  who  were  later  diagnosed  with  breast  cancer.  Facilities  and 
radiologists  are  identified  on  the  confidential  reports  by  MTR  assigned  code  only;  no  facility 
or  radiologists’  names  appear.  Because  of  the  sensitive  nature  of  the  material,  the  reports 
come  to  you  in  an  envelope  stamped  confidential. 

This  individual  radiologist’s  report  can  be  produced  only  after  we  receive  the  signed 
Radiologist  Data  Release  form.  Please  read  and  sign  the  enclosed  form  and  return  it  in  the 
envelope  provided.  After  data  is  received  from  the  facility,  undergoes  a  complete  validation 
process,  and  is  linked  with  the  Washington  State  Cancer  Registry,  you  will  begin  receiving 
the  confidential  Radiologist’s  Quality  Improvement  Report. 

For  mammography  facilities,  the  MTR  can  provide  a  summary  administrative  report  that 
helps  satisfy  the  MQSA  outcomes  and  audit  recommendation.  This  facility  level  report 
contains  no  patient  or  physician  identifiers. 

If  you  have  any  questions  about  the  CQI  program  or  about  the  individual  radiologist’s  report, 
please  call  (800)  224-233  lor  (206)  667-6560  to  speak  with  someone  on  our  staff. 

Thank  you.  We  will  telephone  soon  to  schedule  the  first  CQI  session. 

Sincerely, 


Nicole  Urban,  ScD 
Principal  Investigator 

Enclosures:  Radiologist’s  Data  Release  Form 

MQIP  Timeline 

MTR  Brochure  00064 

Stamped  Return  Envelope 
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September  18,  1997 


Letter  5b 

Notification  of  no  Eandon- 
ization 

lt\A~fz>ducHaM^  t2>  MTT^ 


«cpersfnm»  «cperslnm»,  «etitle» 

«caddrl» 

«caddr2» 

«ccity»,  «cstate»  «czip» 

Dear  Dr.  «cperslnm»: 

Thank  you  for  participating  in  the  Mammography  Quality  Improvement  Project  (MQIP). 
You  were  not  seleeted  to  participate  in  the  randomized  Continuous  Quality  Improvement 
(CQI)  film-reading  intervention  group.  We  will  notify  you,  however,  when  the  CQI 
program  is  available  on  the  Internet.  The  technologists  at  «cgrpnm»  will  be  contacted 
about  their  opportunity  to  attend  a  Continuing  Medical  Education  (CME)  session  at  no 
cost  to  the  facility  or  the  technologist. 

You  may  also  request  information  from  the  Mammography  Tumor  Registry  (MTR).  The 
benefits  of  the  MTR  are  outlined  in  the  enclosed  brochure.  For  the  radiologist,  the  MTR 
can  provide  a  Radiologist  Quality  Improvement  Report.  These  reports  identify  women 
for  whom  you  read  a  mammogram  who  were  later  diagnosed  with  breast  caneer. 

Facilities  and  radiologists  are  identified  on  the  confidential  reports  by  MTR  assigned 
code  only;  no  facility  or  radiologists’  names  appear.  Because  of  the  sensitive  nature  of 
the  material,  the  reports  come  to  you  in  an  envelope  stamped  confidential. 

This  individual  radiologist’s  report  can  be  produced  only  after  we  receive  the  signed 
Radiologist  Data  Release  form.  Please  read  and  sign  the  enclosed  form  and  return  it  in 
the  envelope  provided.  After  data  is  received  from  the  facility,  undergoes  a  complete 
validation  process,  and  is  linked  with  the  Washington  State  Cancer  Registry,  you  will 
begin  receiving  the  eonfldential  Radiologist’s  Quality  Improvement  Report. 

For  mammography  facilities,  the  MTR  can  provide  a  summary  administrative  report  that 
helps  satisfy  the  MQSA  outcomes  and  audit  recommendation.  This  facility  level  report 
contains  no  patient  or  physician  identifiers. 

If  you  have  any  questions  about  the  CQI  program  or  about  the  individual  radiologist’s 
report,  please  call  (800)224-233  lor  (206)667-6560  to  speak  with  someone  on  our  staff. 

Thank  you. 

Sincerely, 


Nicole  Urban,  ScD 
Principal  Investigator 


Enclosures:  Radiologist’s  Data  Release  Form 

MTR  Brochure 
Stamped  Return  Envelope 
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Mammography  Tumor  Registry 
Phone  Contact  A 


Form  2 

Phone  Contact  Script 
-  Data  Collection  ^fethod 
Card  Returned 


Date:  /  / 

Caller: 


Facility  Name 
Address 
Phone  Number 
Contact  Person 


Data  Collection  Card  Returned? 

Contact  Notes: 


yes  (after  introduction,  go  to  section  1) 
no  (after  introduction,  go  to  section  2) 


INTRODUCTION 

Hello.  My  name  is  [Marcia  Gaul/Susie  Wilson],  I’m  calling  from  the  Fred 
Hutchinson  Cancer  Research  Center.  Is  the  facility  manager  available  to  speak  to 
me  at  this  time? 

1.  Yes:  Thank  you. 

2.  No:  Arrange  to  leave  a  message  or  call  back  at  a  better  time. 

Contact  Name:  _ 

Title:  _ 

Phone  Number:  _ 

Today’s  date:  _ _ 

Date  to  call  back: 


SECTION  1 :  Facilities  that  returned  the  data  collection  card 


Hello.  My  name  is  [Marcia  Gaul/Susie  Wilson]  and  I  work  at  the  FHCRC.  I’m 
calling  about  the  Mammography  Tumor  Registry,  a  research  project  we  are 
conducting  here  at  the  Center.  This  is  a  follow-up  to  the  letter  we  mailed  your 
facility  a  couple  of  weeks  ago. 

You  may  recall  that  the  letter  introduced  the  Mammography  Tumor  Registry  which 
provides  follow-up  mammography  Information  to  facilities  and  offers  researchers 
the  opportunity  to  gain  insight  into  issues  related  to  breast  cancer  detection. 
Thank  you  for  promptly  returning  the  reply  card  indicating  how  your  facility  collects 
and  stores  patient  and  exam  information. 

1 .  You  indicated. ..  [restate  information  on  card  and  make  certain  it  is  complete]. 
Photocopy  of  data  collection  card  attached. 


Paper 

_ Patient  Information 

_ Health  History 

Exam  Information 


Computer 

_ Patient  Information 

_ Health  History 

Exam  Information 


Which  database  system  do  you 
NO  CHANGES  use? _ 


2.  We  will  be  sending  a  packet  of  detailed  information  about  the  project,  including 
forms  that  have  to  be  signed,  or  checked  for  accuracy  and  returned  to  us. 

Who  should  receive  this  mailing?  (If  the  radiologist  listed  as  responsible  for 
MQSA  QA,  confirm  spelling  and  address).  (If  someone  else  at  facility,  get 
spelling,  title,  and  address). 


Name: 

Address: 


3.  I  want  to  be  sure  that  we  are  making  the  appropriate  contacts  within  your 
facility,  so  I’d  like  to  ask  if  your  clinic  is  associated  with  any  other  sites  or 
locations? 


1 .  Yes:  a. 

b. 

c. 

d. 


2.  No,  this  is  the  only  location.  (gotoQS) 


4.  Does  each  site  maintain  a  separate  computer  database  of  patient  records,  or 
are  all  records  stored  in  one  main  database? 

1 .  Each  site  has  a  separate  data  base. 

2.  There  is  one  main  data  base. 


Comments: 


5.  Is  there  a  primary  business  office? 

1.  Ves  (go  to  Q6). 


2.  No,  this  is  the  only  office,  (go  to  Q7) 


6.  Who  is  the  database  manager  In  the  business  office?  After  the  packet 
materials  are  returned  to  us,  a  data  staff  person  from  the  project  will  be  in 
touch  with  that  contact  person. 


Contact: 


Phone: 


7.  Who  in  your  office  is  the  database  manager?  After  the  packet  materials  are 
returned  to  use,  a  data  staff  person  from  the  project  will  be  in  touch  with  that 
contact  person. 


Contact: 

Phone: 


Thank  you  for  your  time  today.  The  packet  of  information  will  reach  you  within  the 
next  few  weeks.  We  look  fonward  to  working  with  you. 


Good-bye. 


Mammography  Tumor  Registry 
Phone  Contact  A 


Form  3 

Phone  Contact  Script 
-  Data  Collection  Method 
Card  Not  Returned 


Date:  /  / 

Caller:  _ 


Facility  Name 
Address 
Phone  Number 
Contact  Person 


Data  Collection  Card  Returned? 

Contact  Notes: 


yes  (after  introduction,  go  to  section  1) 
no  (after  introduction,  go  to  section  2) 


INTRODUCTION 

Hello.  My  name  is  [Marcia  Gaul/Susie  Wilson],  I’m  calling  from  the  Fred 
Hutchinson  Cancer  Research  Center.  Is  the  facility  manager  available  to  speak  to 
me  at  this  time? 

1.  Yes.'  Thank  you. 

2,  No:  Arrange  to  leave  a  message  or  call  back  at  a  better  time. 

Contact  Name:  _ 

Title:  _ 

Phone  Number:  _ 

Today’s  date:  _ 

Date  to  call  back:  _ 


SECTION  2:  Facilities  that  did  not  return  the  data  collection  card 

Hello.  My  name  is  [Marcia  Gaul/Susie  Wilson]  and  I’m  calling  from  the  FHCRC.  I 
am  phoning  about  the  Mammography  Tumor  Registry,  a  research  project  being 
conducted  here  at  the  Center.  This  call  is  a  follow-up  to  a  letter  we  mailed  your 
facility  a  couple  of  weeks  ago. 


1 .  Do  you  recall  receiving  a  letter  from  us?  A  brochure  and  reply  card  were 
enclosed  with  the  letter. 

1.  yes  (go  to  Q3) 

2.  no,  confirm  the  address  ( then  go  to  Q2) 

Address: _ 


2.  The  letter  Introduced  a  research  study  called  the  Mammography  Tumor 
Registry  which  Is  being  conducted  by  the  Fred  Hutchinson  Cancer  Research 
Center.  In  sumrnary,  the  Mammography  Tumor  Registry  provides  follow-up 
mammography  information  to  facilities  and  offers  researchers  the  opportunity 
to  gain  insight  into  issues  related  to  breast  cancer  detection.  I  would  be  happy 
to  send  you  another  copy  of  the  letter  and  the  informational  brochure.  Who 
should  receive  this  mailing?  After  you’ve  had  the  opportunity  to  read  the 
information,  I’ll  call  you  again  and  answer  your  questions  and  get  some 
information. 


Name: 

Address: 


3.  I’m  glad  you  received  the  mailing.  I’d  just  like  to  remind  you  that  the 
'  Mammography  Tumor  Registry  is  designed  to  provide  follow-up  mammography 
information  to  facilities  and  it  offers  researchers  the  opportunity  to  gain  insight 
into  issues  related  to  breast  cancer.  To  save  you  the  trouble  of  filling  out  and 
returning  the  postcard,  I  can  take  that  information  over  the  phone.  How  does 
your  facility  collect  and  store  patient  information,  health  history,  and  exam 
information?  (check  computer,  paper,  or  both  for  each) 

Paper  Computer 

Patient  Information _ _  _ 

Health  history  _  _ 

Exam  information  _  _ 

Which  database  system  do  you 
use? _ 


Of  fe*es,  rd  „Ke  ,o  asK  .,our 


or  locations? 
1-  Ves: 


a. 

b. 

c. 

d. 


"o.fft&fetteoa/ytocafoa.  ,gotoQ6) 


records,  or 


Comments: 


2  ff°*®'^of’®®®«®'’«'»fedate6ase 
2.  There  ,s  one  main  data  base. 


6. 


Is  there  a  primaor  business  office? 


■|.  Ves; 


2.  /Vo,  f/7/s  /s  the  only  office.  (Go  to  Q8) 


7-  Who  in  the  business 
information  or  database. 


office  Is  responsible  for 


maintaining  the  patient 


Contact: 
Phone: 


Will  be  in  touch  with  that  contact  person  ’  ^  Project 


Contact: 

Phone: 


«  •  iv-ri  ic;.  - - - - 

in  ffie 


Mammography 
Tumor  Registry 


Form  4 

Facility  -  Data  Release/ 

Confidentiality  Agree¬ 
ment 


HUTCHINSON 

CANCER 

RESEARCH 

CENTER 


DATA  RELEASE/CONFIDENTIALITY  AGREEMENT 


The  information  exchanged  between  participating  radiology  facilities  and  the  Mammography  Tumor 
Registry  (MTR)  is  of  a  highly  sensitive  nature,  including  confidential  patient,  radiologist,  and  cancer 
diagnosis  data.  It  is  absolutely  essential  to  ensure,  to  the  extent  possible,  that  uses  of  MTR  information  is 
limited  to  internal  improvement  in  the  understanding  of  mammography.  Uses  for  any  other  reason, 
particulariy  those  resulting  in  personal  disclosure,  are  strictly  prohibited  by  law. 

The  MTR  and  the  radiology  facility  named  below  mutually  agree  to  the  following: 

1 .  The  radiology  facility  agrees  to  provide  patient-identified  mammography  data  to  the  MTR  for  research 
purposes.  MTR  investigators  and  staff  will  take  all  reasonable  precautions  to  ensure  that  the 
information  provided  to  the  MTR  will  remain  confidential  at  all  times.  Fred  Hutchinson  Cancer 
Research  Center  agrees  to  use  and  disclose  the  information  solely  for  research  relating  to  the  MTR. 
No  facility-identified  information  will  be  disclosed  for  any  reason  without  the  consent  of  the 
participating  radiology  facility.  All  data  contained  in  the  MTR  database  is  protected  from  disclosure  or 
subpoena  by  a  federd  Certificate  of  Confidentiality  issued  under  section  301  (d)  of  the  Public  Health 
Service  Act  by  the  U.S.  Department  of  Health  and  Human  Services. 

2.  The  radiology  facility  will  take  all  reasonable  precautions  to  ensure  that  all  information  provided  by  the 
MTR  to  the  radiology  facility  In  the  form  of  feedback  reports  and/or  data  will  remain  confidential  at  all 
times.  The  facility  agrees  to  use  the  Information  for  quality  Improvement  purposes  only.  Including 
providing  audit  statistics  to  MQSA  and/or  the  ACR,  and  agrees  not  to  publish  or  disclose  the 
information  to  any  other  person  or  entity.  All  information  provided  by  the  MTR  will  be  considered  a 
confidential  medical  record  and  Is  subject  to  Washington  State  law. 


MTR 


Nicole  Urban,  ScD 


Name 


/lfC€rCc 


Signature 


Date 


Principal  Investiaator  -  MTR _ 

Title 

1 100  Fain/iew  Ave  N,  Seattle  WA  98109 
Address 


Radiology  Facility 


Radiology  Facility 


Address 


Name  (print) 


Title 


Signature 


Date 


Fred  Hutchinson  Cancer  Research  Center,  1 100  Faitview  Ave  N.  MP-804,  PC  Box  19024,  Seattle  WA  98109-1024 


MTR  RELEASE  FORM 


■  Form  5 

Facility  Staff  Directory 

Mammography  Tumor  Registry 
Staff  Directory 

The  following  is  the  most  current  personnel  information  that  we  have  for  your  facility.  Please  review 
this  information,  make  corrections  and  additions  as  necessary,  and  return  it  in  the  envelope  provided. 

Facility  Name: 

Mailing  Address: 

Phone  Number: 

Key  Contact  Person:  _ _ _ _ _ 

Position:  _ _ _ _ _ 

Phone  Number:  ( _ )  - 

Which  staff  member  will  provide  the  MTR  with  patient  and  exam  data  files  or  forms? 

Example:  work  with  MTR  staff for  initial  preparation  of  datafile,  data  documentation,  etc. 

^ame:  Phone  Number:  ( _ )  - 

Which  staff  member  will  review  the  data  validation  reports  and  make  corrections? 

Examples:  different  patient  identification  number  assigned  to  same  woman,  discrepancies  in 
assessment  and  follow-up  codes,  etc. 

^ame:  Phone  Number:  ( _ ) 

Who  is  to  receive  the  Mammography  Outcomes  Report? 

This  report  summarizes  by  assessment  the  mammogr(q>hy  performed  at  your  facility  and  the 
results  of  linking  with  the  cancer  registry. 

Name:  _ _  Position:  _ 

Who  is  the  radiologist  responsible  for  MQSA  audit  analyses? 

^ame: _ Phone  Number:  ( _ 

Mammography  Technologist(s): 

_  ID: _ 

• _ ID: _ 

ID: _ 

_ ID: _ 

_ ID: _ 

_ ID: _ 

_ ID: _ 

ID: _ 

ID: _ 

_ ID: _ 

_ ID: 

_ ID: _ 

ID:  _ 


facstdir.frx  -  09/18/97 


Form  6 

Radiologist  Directory 

Mammography  Tumor  Registry 
Radiologist  Directory 


Physician(s): 

Name: 

Mailing  Address: 
Phone  Number: 


Name: 

Mailing  Address: 
Phone  Number: 


Name: 

Mailing  Address: 
Phone  Number: 


Name: 

Mailing  Address: 
Phone  Number: 


Name: 

Mailing  Address: 
Phone  Number: 


Name: 

Mailing  Address 
Phone  Number: 


facradir.frx -05/01/97 


Form  7a 

Questions/Answers  a3x)Ut  ITER 
-Corputer-based  Facility 


The  Mammography 
Tumor  Registry 


Thank  you  for  your  interest  in  the  Mammography  Tumor  Registry  (MTR). 
Below  are  some  questions  and  answers  that  will  be  helpful  as  you  begin. 


What  is  the  Mammography  Tumor 
Registry  (MTR)? 

The  Mammography  Tumor  Registry  is  a 
research  project  that  links  patient  and  exam 
data  provided  by  mammography  faciiities  to 
cancer  incidence  data  collected  by  the 
Washington  State  Cancer  Registry  and  the 
Cancer  Sun/eillance  System.  This 
combination  of  data  enables  the  MTR  to 
provide  participating  facilities  with  summary 
information  regarding  breast  cancer  rates 
among  their  mammography  patients.  In 
addition,  these  data  create  a  rich  pool  of 
information  for  research  investigation  into 
breast  cancer. 

What  kind  of  data  will  my  facility  be 
asked  to  provide  to  the  MTR? 

The  data  you  contribute  must  include 
sufficient  patient  information  to  identify  your 
patients  in  other  databases.  Specificaliy,  this 
includes  the  patient’s  full  name,  birth  date. 
Social  Security  number,  and  current  address. 

Since  your  data  wiil  be  pooled  with  data  from 
other  mammography  facilities,  it  must  contain 
similar  information  to  the  data  provided  by 
those  facilities.  For  example,  each  facility  is 
asked  to  follow  the  ACR  BIRADS  assessment 
coding  system  of  0-5.  If  your  facility  does  not 
use  the  ACR  lexicon  or  uses  a  modified 
system,  we  will  work  with  you  to  understand 
how  your  coding  can  be  made  compatible 
with  the  pooled  data. 


What  staff  responsibilities  are  incurred 
as  we  participate  in  the  MTR? 

Participation  in  the  MTR  requires  some  staff 
time  and  effort,  but  it  should  not  impact  your 
staff  adversely.  A  staff  person  from  the  MTR 
will  work  with  your  facility  staff  person  to 
familiarize  you  with  the  following; 

•  Submission  of  your  database  file  to  the 
MTR  on  a  regular  basis; 

•  Resolution  of  data  discrepancies  on  the 
MTR  validation  form. 

Our  experience  suggests  that  your  initial 
contact  is  likely  to  take  several  hours  over  a 
week  or  two  to  prepare  the  initial  data  and  go 
through  the  first  validation  report.  Once  the 
procedure  becomes  familiar,  MTR  related 
tasks  should  only  take  a  few  minutes  a 
month.  It  is  our  hope  that  the  patient  follow¬ 
up  information  you  receive  from  the  MTR  will 
reduce  the  amount  of  time  you  currently 
spend  on  patient  follow-up. 

How  will  MTR  participation  directly 
benefit  my  facility? 

Following  a  data  validation  process  and  the 
linking  of  your  mammography  data  with 
cancer  incidence  data,  the  MTR  will  send  you 
a  Mammography  Outcomes  Report.  The 
report  offers  summary  statistics  of  your 
facility’s  mammography  results  and  any 
linked  cancers,  and  provides  the  information 
needed  to  calculate  positive  predictive  value. 


computerform 


Form  7b 

Questions/Answers  about  MTR 
-Paper  Facility 


The  Mammography 
Tumor  Registry 


Thank  you  for  your  interest  in  the  Mammography  Tumor  Registry  (MTR). 
Below  are  some  questions  and  answers  that  will  be  helpful  as  you  begin. 


What  is  the  Mammography  Tumor 
Registry  (MTR)? 

The  Mammography  Tumor  Registry  is  a 
research  project  that  links  patient  and  exam 
data  provided  by  mammography  facilities  to 
cancer  incidence  data  collected  by  the 
Washington  State  Cancer  Registry  and  the 
Cancer  Surveillance  System.  This 
combination  of  data  enables  the  MTR  to 
provide  participating  faciiities  with  summary 
information  regarding  breast  cancer  rates 
among  their  mammography  patients.  In 
addition,  these  data  create  a  rich  pool  of 
information  for  research  investigation  into 
breast  cancer. 

What  kind  of  data  will  my  facility  be 
asked  to  provide  to  the  MTR? 

The  data  you  contribute  must  include 
sufficient  patient  information  to  identify  your 
patients  in  other  databases.  Specificaiiy,  this 
includes  the  patient’s  fuil  name,  birth  date, 
Social  Security  number,  and  current  address. 

Since  your  data  wili  be  pooied  with  data  from 
other  mammography  facilities,  it  must  contain 
similar  information  to  the  data  provided  by 
those  facilities.  For  exampie,  each  facility  is 
asked  to  follow  the  ACR  BIRADS  assessment 
coding  system  of  0-5.  If  your  facility  does  not 
use  the  ACR  lexicon  or  uses  a  modified 
system,  we  will  work  with  you  to  understand 
how  your  coding  can  be  made  compatibie 
with  the  pooled  data. 


As  you  begin  participation,  you  are  asked  to 
consider  the  extent  of  the  information  you  can 
provide.  Because  there  is  a  growing  trend  in 
the  research  community  to  investigate  risk 
factors  and  their  relationship  to  early 
detection  and  disease  progression,  we  will 
ask  you  to  collect  detailed  health  history 
information  from  your  patients.  This 
information  may  be  more  extensive  than  that 
which  you  currently  collect. 

How  will  participation  in  the  MTR 
impact  the  facility  staff? 

Participation  in  the  MTR  requires  some  staff 
time  and  effort,  but  it  should  not  impact  your 
staff  adversely.  We  first  ask  that  you  identify 
personnel  who  can  assist  with  the  following: 

•  Collection  of  detailed  patient  and  exam 
information  using  a  standardized  form 
provided  by  the  MTR; 

•  Routine  submission  of  one  copy  of  this 
data  collection  form  to  the  MTR; 

•  Resolution  of  data  discrepancies  on  the 
patient  information  and  exam  form. 

Facility  staff  time  will  involve  monthly 
submission  of  the  patient  information  and 
exam  forms  to  the  MTR.  Additional  staff  time 
may  be  required  to  check  and  resolve  any 
data  discrepancies  we  discover.  Once  the 
procedure  becomes  familiar,  the  MTR  related 
tasks  should  only  take  a  few  minutes  a 
month.  It  is  our  hope  that  the  patient  follow- 
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Mammography  Tumor  Registry 
Review  of  Assessments  /  Follow  Up  Recommendations 
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PATIENT  INFORMATION 


Patient  ID  or  file  number: , 


Form  9 

Patient  History/Exam  Form 


Social  Security  Number _ ■ - - - Telephone  Number - 

First  Name _ Last  Name _ Middle  Initial -  Date  of  Birth 


t  Street  Address. 


-City. 


.  State. 


Zip. 


ETHNIC  BACKGROUND 
□t  Caucaslan/White 
□2  African  American/Black 
□3  Native  Amerlcan/Esklmo/AIeut 
□4  Asian/Pacific  Islander 
□e  Other 

HISPANICAATINA  ORIGIN  Do  No 

HISTORY 


□1  Yes 


EDUCATION  (chock  only  one) 

□t  1-11  Years 
□2  High  school  graduate 
□3  Some  collegeAechnIcal  school 
□4  College  graduate  (4  years) 

□5  Post  graduate  degree 


HEALTH  INSURANCE  (check  all  that  apply) 
□1  None  ^ 

□1  Medicare 
□1  Medicaid 

HMO,  Managed  Care 
□1  Private  Insurance  Company 
□1  Other 
□1  Not  Sure 


2.  Has  vour  mother  had  breast  cancer? 


3.  How  many  of  your  sistsrs  had  breast  cancer? 
□a  I  have  no  slaters 
ft  None  of  my  slaters 


^^y^^^jr^terB:underege  50. 


1 


4.  How  many  of  your  daughters  had  breast 
cancer? 

□1 1  have  no  daughters 


□2  Two  or  mors  daughters  — 
□9  Not  sure  ^ 

c _ 

If  Jyti,^sre  any  of  your  daughters 
ij^OTdlag^ed?-"  .  - 

□2  Yes^two  or  more  daughters 
□9j^oJ.8ure 

under  age 

6.  Has  any  relative  had  ovarian  cancer? 

□oNo 

□1  Mother,  sister  or  daughter 
□2  Aunt  or  grandmother 
□3  Other  relative 
□a  Not  sure 


6.  Previous  breast  procedures 

(check  all  that  apply) 


Rne  Needle  Aspiration 
Core  Needle  Biopsy 
Open  Excislonal  Biopsy 
Lumpectomy 
Mastectomy 
Radiation  Therapy 
Reconstruction 
AugmentatlorVImplants 


7.  Date  of  most  recent  breast  biopsy: 

/  /  Di  Never  had  a  biopsy 

8.  Your  age  at  the  birth  of  your  first  child: 

_ _  □»  I  have  no  natural  children 

9.  Have  your  menstrual  periods  stopped 
permanently?  (^eck  only  one) 


Left 

Right 

Both 

□t 

Dz 

□a 

□t 

□2 

□3 

□1 

□2 

□3 

□1 

□2 

□3 

□1 

□2 

□3 

□1 

□2 

□3 

□1 

□2 

□a 

□1 

□2 

□a 

rertac^^t 


O4^yes.vmyperioda.8topped*^ 

:-'>4dUeWsurgery^^^ 

'□^NoPtufi*' 


If  yeSf  how  old  were  you  when  your 
perioda  stopped? 


If  no,  Is  the  approximate  length  In  days  of 
■^Mr3>«wtni^;cycle?'-^^^ 

Ancl^^ati^  the  date^^^^^^  last  v ; 

menstrual  cycle  (ple^e  estimate  If  you  don't  know 
the  exact  rfay)  /  •  /  _ _ 


10.  Have  you  had  one  or  both  ovaries 
removed? 

□oNo 

Ot  Yes,  one  ovary  removed 
□2  Yes,  two  ovaries  removed 
□9  Not  sure 

DO  NOT  WRITE  BELOW  THIS  LINE 


11.  Are  you  currently  using  any  hormones? 
(check  all  that  apply) 

□t  No 

□1  Yes,  Estrogen  only 

□1  Yes,  Estrogen  and  Progesterone 

□t  Yes,  Tamoxifen 

□t  Yes,  birth  control 

□1  Yes,  other  hormone 

□1  Not  sure 

12.  Have  you  had  any  pi’oblems  or  symptoms 
with  your  breasts  In  the  last  3  months? 
□oNo 
:lf^;iche 

Ltpip' 

Nipple  discharge 

Parri'-^  -  .:  ■  4  ^^-’ 

Skinrchanges; 

Other 

13.  Did  you  make  this  appointment  due  to  a 
concern  about  a  breast  problem  found  In  the 
past  3  months?  (check  one) 

□0  No,  this  Is  a  routine  mammogram. 

□1  Yes,  I  found  something  new. 

□2  Yes.  my  doctor  found  something  now. 

□3  Yes,  I  have  a  general  concern  but  no 
specific  symptoms. 

14.  Have  you  had  a  previous  mamfnogram? 
□oNo  DiYes  □«  Not  sure 

— —  'i'  — - — 

If  yes,  v^at.was  the  date  of  your  last% 
marnmog^V^'-  > 


16.  Have  you  ever  had  a  clinical  breast  exam 
(a  physical  breast  exam  performed  by  a  health 
care  provider)? 

□0  No  □nYesi  □»  Not  sure 

M 


Ifyes,  how  long  since  younlMt  _ 
breast  exam?  (Check  only  one) 

□t  Within  the  last  3  months 

■  [^IMore  ^an  d'^year.ago!|^.^^^^?g^ . 


EXAM  INFORMATION  (To  be  completed  by  clinic  personnel) 


Factltty/Exposurs  Sit*  _ 


T*eh  ID  _ 


Radiologist  ID  . 


Data  of  Mammogram  _ 


1 .  Physical  txam  rotulu 
□0  Nagattvo 

□,  PosHh/a  (auapidoua  for  malignancy) 
□t  Not  performed 

2.  Symptoms  (check  all  that  apply) 

.  Di  None 

□1  Lump 

□t  Bloody  nippla  discharge 
□1  Psin 

□t  Other _ _ _ - 


3.  Waa  patient  referred  beoatia*  of  aymptoma  detected 
by  CBE  performed  within  the  last  3  months? 

□oNo  DiYea  □«  Unknown 

4.  Date  of  last  mammogram  - / - 1 - 

8.  Comparison  fllma  available?  Do  No  QiYes 

e.  Reason  for  mammogram 
(chackonlyona) 

□0  Screening  (asymptomatic) 

□t  DIagnoetIc  (symptomatic) 

□t  Short  Interval  follow-up 
□,  Additional  viow(i)  for  current  exam 
□4  Special  study 

□1  Other: _ — - 

7.  Procedure 

□]  BflaterW  mammography 
□2  Right  only 
□1  Left  only 


8.  Density  (code  breast  with  greatest  density) 

□,  Mostly  fatty 

□2  Scattered  fibrogiandular  tissue 
□2  Heterogeneously  dense 
□4  Extremely  dense 

9.  Aaeaaament  •  Right  Breast 
□0  Needs  additional  evaluation 
□,  Normal 

□2  Benign  hndirtg 
□,  Probably  benign;  short  follow-up 
□4  Suspicious  abnormality 
□,  Highly  auspJdous  for  malignancy 

10.  Ataaaament  -  Left  Breast 

□2  Needs  additional  evaluation 

□,  Normal 

□2  Benign  finding 

□j  Probably  benign:  short  follow-up 

□4SU8PICIWS  abnormality 

□2  Highly  suspidous  for  malignancy 

11.  Aaseseme nt  baaed  on:  (ehed(  all  that  apply) 

Basic  2  views  per  breast 

□1  Additional  ^ews 

□,  Left  breast  Right  breast  □,  Both  breasts 
□,  Clinical  findings 
□,  Referring  physlden’s  report 
□i  Comparison  with  previous  films 
□,  Patient  report 
nt  Ultrasound 
□t  FamHy  history 
□,  Patient  history 


12.  Recommendation  for  mammogram  follow-up 

□t  Routine  follow  up  Interval  Months: _ 

□t  Short  term  follow  up  Months: _ 

13.  Recommendation  for  immediate  wortt-up 
(check  all  that  apply) 

□,  Additional  views 
□1  Ultrasound 
□tFNA 

n,  Core  needle  biopsy 
□,  Surgical  biopsy 
□,MRI 

□,  Surgical  or  clinical  consult 

□,  Other  Immediate  workup:  - - 


Signature 
O  » lump 


-scar 
X  -pain 


i 
I 


Form  11 

MQIP  Consent  to  Participate 
Radiologist 


Mammography  imi 

Quality  IrI 

Improvement 
Project 

RADIOLOGIST’S  PARTICIPATION  AGREEME 


hKbU 

HUTCHINSON 

CANCER 

RESEARCH 

CENTER 


NT 


Investigators;  Nicole  Urban,  ScD,  Principal  Investigator,  (206)  667-6560 

Mariann  Drucker,  MD,  Radiologist,  (206)  667-6560 
Connie  Lehman,  MD,  PhD,  Radiologist,  (206)  543-3320 

All  work  conducted  by:  Fred  Hutchinson  Cancer  Research  Center 

Cancer  Prevention  Research  Program 
1 100  Fairview  Ave.  N  (MP  804) 

PO  80x19024 
Seattle,  WA  98109-1024 


Background: 

Breast  cancer  is  one  of  the  major  causes  of  death  among  women  age  50  or  older.  At  least  one  in  nine 
women  in  the  United  States  will  be  diagnosed  with  breast  cancer  in  her  lifetime.  Despite  research  efforts 
in  breast  cancer  prevention,  a  woman’s  best  hope  for  cancer  survival  lies  in  identification  of  the  disease 
early  enough  to  treat  it  effectively.  The  primary  tool  for  early  detection  is  regular  screening  mammography. 
In  order  to  improve  the  quality  of  screening  mammography,  it  is  hypothesized  that  a  Continuous  Quality 
Improvement  (CQI)  program,  which  provides  radiologists  with  immediate  feedback  on  their  interpretation 
of  a  set  of  mammograms  selected  for  their  teaching  value,  can  significantly  improve  the  ability  of  the 
radiologist  to  detect  early  stage  breast  cancer. 

Purpose: 

We  are  asking  you  to  consent  to  participate  in  a  randomized  study  being  conducted  within  a  research 
project,  the  Mammography  Quality  Improvement  Project  (MQIP).  The  primary  research  objective  is 
to  determine  if  a  Continuous  Quality  Improvement  (CQI)  program  can  increase  the  accuracy  of 
mammography  film  interpretation  by  radiologists.  The  secondary  research  objectives  are  to  1) 
determine  inter-rater  variability  in  film  interpretation,  before  and  after  implementation  of  the  CQI  program, 
in  a  set  of  mammograms  selected  for  their  teaching  value;  2)  determine  whether  digitized  films  can  be 
interpreted  with  the  same  accuracy  as  high-quality  copies  of  films;  and  3)  determine  if  the  accuracy  of  film 
interpretation  depends  on  covariates  such  as  the  woman's  age  and  breast  density. 

Procedures: 

You  were  selected  for  participation  in  this  study  because  you  regularly  read  mammograms  for  a 
Washington  State  mammography  facility  that  is  part  of  the  Mammography  Tumor  Registry.  These 
facilities  will  be  rank  ordered  by  the  number  of  radiologists  reading  mammograms  for  each,  and  then 
paired.  Randomization  to  the  CQI  program  group  or  to  the  control  group  will  take  place  within  these  pairs. 
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Accordingly,  you  have  a  50%  chance  of  being  randomized  to  participate  in  the  CQI  program.  The  CQI 
program  procedures  apply  to  you  only  if  you  are  randomized  to  the  film  reading  study. 

The  CQI  program  consists  of  five  sessions  and  involves  reading  a  total  of  180  mammography  films 
(cases)  randomly  assigned  to  five  sets  -  2  sets  of  45  films  and  3  sets  of  30  films.  Radiologists  are 
randomly  assigned  sets  of  films  to  be  read  at  each  of  the  sessions.  Scheduled  over  a  12  - 1 4  month 
period,  the  five  two  hour  film-reading  sessions  will  be  conducted  individually,  on-site,  and  will  be  facilitated 
by  a  project  field  coordinator.  Sessions  1-4  will  utilize  both  films  and  digitized  images.  Session  5  offers 
only  digitized  images  in  order  to  determine  the  feasibility  of  CQI  program  dissemination  via  the  Internet. 

For  each  session,  an  appointment  will  be  a  made  for  you  to  read  a  set  of  films  and  code  your  assessment 
of  each  case  on  a  laptop  computer.  The  films  will  be  high-quality  copies,  because  the  originals  are  not 
available  to  the  project  for  the  requisite  18  month  period.  You  will  be  instructed  to  read  the  films  as  you 
generally  read  screening  mammograms  with  one  exception  -  the  cancer  prevalence  in  the  population 
from  which  the  films  were  selected  is  higher  than  you  would  expect  in  your  practice. 

Your  assessment  will  be  made  using  the  standard  ACR  5-point  scale  as  follows; 

•  1  indicates  “normal,  routine  follow-up  recommended,” 

•  2  indicates  “benign,  routine  follow-up,” 

•  3  indicates  “probably  benign,  early  recall  recommended,” 

•  4  indicates  “suspicious  for  cancer,  consider  biopsy,” 

•  5  indicates  “highly  suspicious  for  cancer,  biopsy  recommended.” 

You  will  also  be  asked  to  indicate  the  suspicious  abnormality  on  the  digitized  computer  image. 

You  must  read  and  assess  all  films  in  the  set  before  the  computer  offers  feedback.  This  feedback 
consists  of  the  correct  interpretation  of  each  film  with  indication  of  the  areas  of  benign  or  malignant 
disease.  Text  will  explain  the  nature  of  disease  (if  any)  and  the  strategy  for  interpretation. 

Film  Solicitation: 

The  180  films  used  in  the  study  will  include  women  with  and  without  cancer,  as  well  as  both  younger  and 
older  women.  We  anticipate  that  up  to  one-half  of  the  films  will  be  contributed  by  the  participating 
radiologists.  As  a  participant,  you  are  asked  to  submit  interesting  films  for  possible  inclusion  in  the  study. 
The  mammograms  must  be  from  women  who  were  asymptomatic  at  the  time  of  the  film.  At  least  one  of 
the  films  should  be  from  a  woman  <50  years  of  age.  Please  submit  films  that  were  challenging  to 
interpret.  Malignant  cases  must  have  biopsy  correlation.  Benign  assessment  is  determined  either  by  a)  a 
woman’s  cancer-free  status  two  years  after  the  mammogram:  b)  additional  evaluation  resulting  in  a  benign 
finding;  or  c)  negative  biopsy  results. 

Other  films  used  in  the  sessions  will  be  selected  from  radiologists  around  the  state  and  will  be 
supplemented  with  interesting  cases  from  the  teaching  files  of  the  radiologist  investigators.  Confidentiality 
regarding  the  source  of  the  film  is  maintained  at  all  times.  Before  a  film  is  reviewed  for  inclusion  in  the 
study,  it  will  be  masked.  There  will  be  no  patient,  radiologist,  or  facility  identifiers  on  any  of  the  study  films. 

Continuing  Medical  Education: 

The  CQI  program  will  be  an  ACR/UW  accredited  Continuing  Medical  Education  (CME)  program.  It  is 
anticipated  that  a  2  hour  appointment  will  be  required  for  each  film  reading  session.  Two  CME  credits  will 
be  awarded  for  each  session,  for  a  total  of  10  CME  hours. 

Risk: 

The  only  possible  risk  involved  in  study  participation  is  the  potential  loss  of  privacy  associated  with  the 
data  describing  performance  on  the  mammography  practice  readings.  To  minimize  this  risk,  each 
radiologist  is  assigned  a  personal  identification  number.  In  the  data  files,  you  are  identified  only  by  this 
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number.  The  information  linking  names  with  numbers  wiil  be  stored  in  seaied  files  at  the  Fred  Hutchinson 
Cancer  Research  Center.  Once  the  necessary  information  is  obtained  from  data  files,  all  identifying 
information  wiii  be  destroyed.  Personal  and  facility  identity  will  not  be  revealed  in  any  pubiication  or  in  the 
results  of  the  study.  Study  records  will  be  maintained  indefinitely  for  the  purpose  of  analysis  and  are 
accessible  only  by  research  staff  and  the  US  Army  Medical  Research  and  Materiel  Command. 

Representatives  from  the  U.S.  Army  Medical  Research  and  Materiel  Command  (and,  where  applicable, 
the  Food  and  Drug  Administration,  and  the  U.S.  Army  Medical  Department  Center  and  School)  may 
inspect  the  records  of  the  research  in  their  duty  to  protect  human  subjects  in  research. 

Other  pertinent  information: 

Participation  is  completely  voluntary.  If  you  agree  to  participate,  you  are  free  to  withdraw  at  any  time.  Your 
refusal  to  participate  or  a  decision  to  discontinue  participation  will  involve  no  penalty  or  loss  of  benefits  to 
which  you  are  othenwise  entitled. 

The  Department  of  Defense  is  funding  this  research  project.  Should  you  be  injured  as  a  direct  result  of 
participating  in  this  research  project  you  will  be  provided  medical  care,  at  no  cost  to  you,  for  that  injury. 

You  will  not  receive  any  injury  compensation,  only  medical  care.  You  should  also  understand  that  this  is 
not  a  waiver  or  release  of  your  legal  rights.  You  should  discuss  this  issue  thoroughly  with  the  Principal 
Investigator  before  you  enroll  in  this  study.  There  is  no  compensation  for  involvement  in  this  study. 

For  further  information  about  the  study  please  contact  the  project  office  at  (206)  667-6560  or  (800)  224- 
2331 .  If  you  have  any  questions  about  your  rights  as  a  study  participant,  please  contact  Karen  Hansen  in 
the  Institutional  Review  Office  of  the  Fred  Hutchinson  Cancer  Research  Center  at  (206)  667-4867. 
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I  have  provided  an  explanation  of  the  above  research  program.  The  subject  was  given  an  opportunity  to 
discuss  the  procedures,  including  possible  alternatives,  and  to  ask  additional  questions.  A  signed  copy  of 
the  consent  form  has  been  given  to  the  subject. 


Investigator  Signature  Date 


Subject’s  Statement 


I  agree  to  participate  in  this  study  (and  to  the  conditions  outlined  in  this  consent  form  that  I  have  read  and 
signed).  I  had  the  opportunity  to  ask  questions  about  the  study  and  my  participation.  All  questions  were 
answered  to  my  satisfaction.  I  understand  that  future  questions  I  may  have  about  the  research  will  be 
answered  by  the  project  investigators  listed  herein  and  that  any  questions  I  have  about  my  rights  as  a 
research  subject  will  be  answered  by  them.  I  give  permission  for  my  study  records  to  be  available  to 
physicians,  scientists,  and  personnel  working  on  this  study  at  the  Fred  Hutchinson  Cancer  Research 
Center.  Records  are  also  available  to  the  US  Army  Medical  Research  and  Materiel  Command.  I 
acknowledge  that  I  will  receive  a  signed  copy  of  this  consent  form.  I  read  and  initialed  each  page  of  the 
form. 


Participant  Signature  Date 


Printed  Name 


Address 


City,  State,  Zip 


Witness  Signature 


Printed  Name 


Address 


City,  State,  Zip 
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Mammography  Quality 
Improvement  Program 

Instructions  for  Case  Selection 


Mammography  Tumor  Registry 

Fred  Hutchinson  Cancer  Research  Center 

1 1 00  Fairview  Ave  N  (MP-804) 

PO  Box  19024 
Seattle.  WA  98109-1024 


Please  choose  mammograms  from  asymptomatic  women  that  offered  you  a  diagnostic  chalienge.  Include 
an  equal  number  of  benign  and  malignant  assessments.  Include  at  least  one  case  from  a  woman  under 
50.  Other  criteria  include: 

•  The  film  is  of  educational  interest  because  interpretation  was  challenging.  Malignant  cases  must  have 
biopsy  correlation.  Benign  assessment  is  confirmed  by  a)  a  woman’s  cancer-free  status  two  years 
after  the  mammogram;  b)  additional  evaluation  with  a  benign  finding;  or  c)  negative  biopsy  results. 

•  Submit  cases  with  only  one  challenging  finding;  exclude  cases  with  either  two  synchronous  cancer 
sites  in  separate  quadrants  of  one  breast  or  cases  with  bilateral  breast  cancer. 

•  Send  no  mammograms  at  this  time.  Fill  out  this  form  completely  and  the  original  films  will  be 
requested  from  your  film  room.  Films  will  be  masked  for  identifiers,  copied,  and  then  seiected.  Original 
films  will  be  returned  within  eight  weeks.  For  emergency  immediate  film  return,  call  (800)  224-2331  or 
(206)  667-6560. 


Patient  Information 


Patient’s  name: _  Medical  ID#; _ 

Prior  name(s)  used  by  the  patient: _ ; _  Date  of  Birth: 

Social  Security  Number: _ - _ - _ 

Case  Information 


Date(s)  of  the  mammogram(s)  to  be  considered: 

Facility  where  mammogram  is  stored: _ 

Areas  of  interest/Description  of  case: _ 


Classification  of  finding(s):  Check  all  that  apply. 

□  Benign/Normal  Mammogram 

□  Calcifications 

□  Masses 

□  Architectural  distortion 


ACR  Assessment 
□i  Normal 
□2  Benign  finding 
□3  Probably  benign;  short  follow-up 
□4  Suspicious  abnormality 
□s  Highly  suspicious  for  malignancy 


Date  of  biopsy:  (if  performed)  /  / 


Biopsy  results: 

□  Benign 

□  Normal  ductal  epithelium 

□  Fibroadenoma 

□  Cyst 

□  Sclerosing  Adenosis 

□  “Fibrocystic  changes” 

□  Other _ 


□  Malignant 

□  DCIS 

□  Invasive  ductal  carcinoma 

□  LOIS 

□  Invasive  lobular  carcinoma 

□  Other _ 


Location  of  finding: 


caseselect 


I  authorize  temporary  release  of  these  films 
I  to  the  Fred  Hutchinson  Cancer  Research 
Center  for  educational  purposes. 


Physician’s  Name  (please  print) 


physician’s  Signature 


